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4 күңе 


ју, INTRODUCTION 


PURPOSE 

The purpose of this thesis is to explore and identify key problems in Developmental 
Testing and Evaluation (DT&E) in the United States Army. A comparative analysis of 
several programs is conducted to determine common developmental testing problems. 
These problems are analyzed and a set of conclusions and recommendations for future 
testing is developed. The thesis provides the reader with a current understanding of the 
Department of Defense (DOD) test structure, its relationship to the acquisition process, and 
the Department of the Army test agencies involved in Developmental Test and Evaluation. 
The thesis concludes by providing recommendations to help future testers, evaluators and 


program managers to better prepare for DT&E in the acquisition life cycle. 


B. BACKGROUND 

Test and Evaluation (T&E) is required in the acquisition life of any Department of 
Defense system. Developmental testing is critical to a program moving on in the 
acquisition process. Developmental testing, like the entire acquisition process is subject to 
budget cycles, agency conflicts, turf battles, congressional influence and other political 
factors all of which can cause problems. Problems in developmental testing have caused 
schedule delays, inadequate or useless data and unplanned expenditures of large amounts 
of money and other resources to resolve the situations. In the current environment of 
declining budgets, such problems can result in program slowdown or even cancellation. 

There are two major categories of test and evaluation: (1) Developmental Test and 
Evaluation (DT&E) and (2) Operational Test and Evaluation (OT&E). Operational Testing 


as well as other types of testing are described, but the focus of this thesis is DT&E. 


Developmental! Test and Evaluation (DT&E) is conducted throughout the acquisition 
process to ensure the acquisition and fielding of an effective and supportable system. Early 
DT&E is normally conducted by the sub-contractors and the prime contractor. The sub- 
contractors test the components as they are developed and the prime is interested in testing 
the total system as the components are integrated. The Government test agencies are more 
involved during the later acquisition phases to demonstrate how well the weapon system 
meets its technical requirements. 

There are several key players involved in DT&E in the U. S. Army. DT&E is 
normally planned, coordinated, conducted and monitored by the United States Army Test 
and Evaluation Command (TECOM), a sub command of the Army Materiel Command 
(AMC). The test design, evaluation and analysis role of DT&E is conducted by the United 
States Army Materiel Systems Analysis Activity (AMSAA). Other critical players include 
the future users or Combat Developers from the United States Army Training and Doctrine 
Command, (TRADOC), and the Program Management Office (PMO) charged with the 


overall acquisition of the system including testing. 


C. OBJECTIVES OF THE THESIS 

The objective of the thesis is to explore and identify common problems of 
developmental testing in the United States Army from the perspective of the different 
agencies involved. The agencies or groups focused on were: the Program Management 
Office (PMO), the testers and their facilities, the analysts/evaluators, the contractors, and 
the users. 

The Program Management Office (PMO) is ultimately responsible for all aspects of 
any program including testing. The PMO normally maintains a section whose functions 
include test coordination. The Army's Test and Evaluation Command (TECOM) provides 


most of the resources -- testers and facilities -- for DT&E in the Army today. Analysis and 


evaluation for Army DT&E is normally conducted by the Army Materiel Systems Analysis 
Agency (AMSAA). This agency impacts the analytical and statistical side of the DT&E. 
The contractor or builder plays a significant role in the test, fix, retest scenario of the early 
stages of a weapon system development. Finally, the users provide the need and request 
the capabilities that the system under test will hopefully provide. The user or Combat 
Developer is usually represented by a TRADOC Systems Manager (TSM) or by a 
combined TRADOC organization such as the Combined Arms Center at Fort Leavenworth, 
Kansas. 

The organization and basic DOD test process 1s described followed by the Army test 
Structure down to the agencies and offices that are the focus of the data collection. Data are 
presented and once the developmental test problems are identified, they are analyzed and a 
set of recommendations is developed from this information. The recommendations can 
then be used by program managers, testers and evaluators to help them assess and prepare 


their system for developmental testing. 


D. RESEARCH QUESTIONS 
The primary research question for this thesis is: 
What is the most significant problem in conducting developmental testing? 
The following are subsidiary research questions to help develop and define the primary 


research question. 
1. What are the common developmental test problem areas across agencies? 
2. What are the common developmental test problem areas across types of systems or 
acquisition strategies? 
3. To what extent do these problems endanger program success? 
4. To what extent do these problems impact cost and schedule? 
5. What can be done by program managers, testers and evaluators or others to 
improve the preparation and conduct of developmental testing? 


E. SCOPE AND LIMITATIONS 

The scope of the research is limited to Acquisition Category (ACAT) I and [0 systems 
tested at United States Army Test and Evaluation Command (TECOM) facilities within the 
United States. Seven programs under test or tested in the past 10 years are researched and 
analyzed. The systems include the Abrams Main Battle Tank Block IJ upgrade (M1 A2); the 
Javelin, a man portable antitank weapon, (AAWS-M); Enhanced Position Location 
Reporting System (EPLRS); the Avenger, a mounted Air Defense system, the Kiowa 
Warrior, an armed scout helicopter; the Maneuver Control System (MCS), a command and 
control system; and the Family of Medium Tactical Vehicles (FMTV). These programs 
represent different types of systems from electronic communications and software to major 
weapon systems and represent different types of acquisitions from system upgrades and 
Non-Developmental Items (NDJ) to full scale development. Other information will come 


from literature searches and interviews as indicated below. 


F. RESEARCH METHODOLOGY 

Research investigation included a literature search of After Action Reports from Test 
and Evaluation Command (TECOM), lessons learned reports of major systems, General 
Accounting Office Reports, Congressional Subcommittee Reports, Developmental Test and 
Evaluation Reports and technical and professional journals. Developmental testing does 
not normally receive the public scrutiny of operational testing and therefore much of the 
information was gathered through the interview process. Interviews were conducted with 
program office personnel, program test officers, analysis personnel, Combat Developers 
and contractors who participated in developmental testing of the major programs already 
mentioned. Supervisors from AMSAA and TECOM with years of DT&E experience in the 
Army were also interviewed. Interviews were conducted in person, over the phone and | 


through the use of video teleconferencing. 


G. ORGANIZATION OF THE STUDY 

The thesis is organized into the following chapters. 

Chapter Il: Background -- This chapter contains historical information on DOD testing 
and describes the current DOD test structure within the acquisition process. The chapter 
then describes the test structure in the U. S. Army focusing on DT&E and the key players 
in Army DT&E. 

Chapter III: Methodology and Data Summary -- This chapter explains the methods 
used for executing the research design and structure of the analysis. The chapter then 
presents the data that were used for the analysis. Most of the data are in the form of 
interviews. 

Chapter IV: Results and Analysis -- This chapter analyzes the data and indicates their 
implications. 

Chapter V: Conclusions and Recommendations -- This chapter contains a summary of 


the principal findings of the thesis and offers recommendations for future use. 


II. BACKGROUND 


A. INTRODUCTION 

This chapter provides a basic history of testing in the Department of Defense (DOD) 
with emphasis on the last twenty years. The chapter describes the four major categories of 
testing namely Development Testing, Operational Testing, Joint Testing and Multi-Service 
Testing. The chapter then describes testing within the acquisition process and concludes 
with the description of the Developmental Testing function within the United States Army. 

Test and Evaluation (T&E) is a critical part of the acquisition process. "Test" denotes 
the actual testing of hardware or software to obtain data. These data are valuable in 
developing new capabilities, managing the process, or making decisions on the allocation 
of resources. "Evaluation" denotes the process whereby data are logically assembled and 
analyzed to aid in making systematic decisions. "Test and Evaluation is the process by 
which a system or components are compared against requirements and specifications 
through testing." [Ref. 1] 

The planning and conducting of T&E exists throughout the acquisition cycle. There is 
the need for thorough, logical, systematic, and early test planning and the feedback of well 
documented, unbiased test and evaluation results to system developers, users, and decision 
makers. The purpose of Test and Evaluation in a defense system's development and 
acquisition program is to identify the areas of risk to be reduced or eliminated. During the 
early phases of development, T&E is conducted to demonstrate the feasibility of conceptual 
approaches, to minimize design risk, to identify design alternatives, to compare and analyze 


tradeoffs, and to estimate operational effectiveness and suitability. [Ref. 2] 


В. НІ5ТОКҮ 
1. World War II to 1960's 

Equipment testing has been a part of the procurement process throughout the 
nation’s history. For decades testing remained informal, generally “ad hoc" and evolved 
along with the procurement process. During World War II the procurement process began 
to take on a more formalized management approach. As the procurement process evolved 
so did testing. Testing was conducted throughout World War IJ. There were engineering 
tests to test engineering and scientific characteristics and Service tests conducted by the 
various branches to determine if the equipment was sufficient for field use. Testing was 
basically sequential, lengthy and dependent on the need of the equipment. 

The war ended with much equipment still in testing and a recognition that 
research, development and evaluation must continue to be a peacetime effort. Those 
involved in testing during the war determined that testing could be greatly simplified under 
more completely integrated development agencies. The development and continued testing 
of the atomic bomb offered a ready example of such an integrated and expedited effort. An 
R&D Division was developed in the Army General Staff in 1946, however it soon fell 
victim to demobilization and by 1948 the functions of the R&D division were assigned to a 
subgroup in the Logistics Division. [Ref. 3] 

Following the war most of the research, development and testing conducted in 
DOD dealt with rockets and missiles. During this time the procurement process was 
evolving and terminology like systems engineering, operations research, project offices and 
contracted engineering support started to come into play. 

2. 1960's and 70's 

The 1960’s saw the formalization of the acquisition process and subsequently the 

formalization of the testing process. Initially Secretary of Defense McNamara took a 


business approach to the acquisition process fostering total package procurement, strong 


centralized civilian control and concurrency. In the early 1970's, Secretary of Defense 
Laird and his deputy, David Packard, promoted decentralization and a “fly before buy” 
mentality. The Department of Defense (DOD) test policy within the acquisition system 
became more formalized and placed greater emphasis on Test and Evaluation (T&E) as a 
continuing function throughout the acquisition cycle. The policies stressed the use of T&E 
to reduce risk and provide a continuous estimate on the system's effectiveness and 
suitability. To meet these objectives it was important that appropriate test activities be fully 
integrated into the overall development process. [Ref. 4] It was also during this time that 
both the Army and the Air Force were given a congressional directive to establish 
independent operational test agencies. 
3. 1980's 

Early in the 1980’s the acquisition system was again reviewed and revised. Test 
and Evaluation was further refined in DOD Directives and Military Standards documents 
and was becoming a major concern of the Pentagon and The Congress. In 1983 Congress 
directed the establishment of the Director Of Operational Test And Evaluation (DOT&E). 


In 1983 the Assistant Secretary Of Defense made the following statement: 


... the criterion should not be how quickly we can field any new weapon, but 
rather how quickly we can field a new weapon that works. The only weapons that 
would be significantly delayed would be the ones that operational testing shows to be 
unsuitable for combat, and I cannot believe that any of us would advocate saddling 
our fighting forces with any of those. In fact, the most likely effect of operational 
testing is not to delay, but to accelerate the development process. Trying to fix a 
faulty weapon after it's in the field -- if 1t can still be fixed -- is a far slower process 
than fixing the design before it goes into production. [Ref. 5] 


Testing continued to be a target of discussion and reform from outside as well as 
inside the Pentagon throughout the 1980's along with the acquisition process. 
4. DOD Acquisition Revision 
Defense Management was again looked at in 1985 by a Presidential Blue Ribbon 


Commission on Defense Management and there were still other revisions of the acquisition 


process in 1987 and in 1991. The focus of each of these revisions was an attempt to make 
the acquisition process less costly, less time consuming, and more responsive to the needs 
of the operational community. [Ref. 6] The result of these latest revisions is our current 
acquisition system, a system that is again under review. Among the many initiatives of the 
latest revisions was the push to consolidate documentation. In the testing arena, DOD 
Directive 5000.3 "Test and Evaluation" was canceled along with other 5000 series 
Directives and integrated into DOD Directive 5000.2, February 1991, part 8. Other affects 
on testing include the requirement of live fire testing of major weapon systems before the 
production phase and the inclusion of test information in various reports, the Selected 
Acquisition Report (SAR) for example. [Ref. 7] The acquisition process is again under 


review and almost any revisions to the acquisition process will likely impact on testing. 


C. TYPES OF T&E 

There are four major types of testing, DT&E (Development), OT&E (Operational), 
Multi-Service Test and Evaluation and Joint Test and Evaluation. The types of testing 
which fall into the realm of DT&E or at least Director, Test and Evaluation (DTE) oversight 
include qualification testing, Live Fire Test and Evaluation (LFT&E) and Production 
Acceptance Test and Evaluation (PAT&E). The following sections describe these various 
types of testing. [Ref. 8] 

1. Development Test and Evaluation (DT&E) 

Development Test and Evaluation (DT&E) is an iterative process of design, build, 
test, identify deficiencies, fix, retest, and repeat. DT&E is conducted throughout the 
acquisition process to ensure the acquisition and fielding of an effective and supportable 
system. It is performed in the factory, laboratory and on the proving ground by contractors 
and the Government. DT&E includes test and evaluation of components and subsystems at 


all Work Breakdown Structure (WBS) levels, it includes modifications, hardware/software 


iterations, related software and qualification testing. Contractor and Government testing 
may be combined into one integrated program test and conducted to determine if the 
technical development of the acquisition process has been met as well as provide data to the 
decision authority. [Ref. 9] DT&E involves the use of simulations, models, breadboards, 
brassboards, and test beds, and full scale engineering development models or prototypes of 
system components or the system itself. DT&E is conducted throughout the acquisition 
process as described in the "Testing and the Acquisition Process" section of this chapter. 
a. Qualification Testing 

Qualification testing is performed to verify the design and manufacturing 
process and provide a baseline for subsequent acceptance tests. These tests include 
Production Qualification Tests (PQT), First Article Tests (FAT) and other down line 
production qualification tests performed to verify process control. The Production 
Qualification Test 1s conducted at the unit, subsystem (component) and system level on 
production items and completed before production decisions. The First Article Tests (FAT) 
consist of a series of formal contractual tests conducted to ensure the effectiveness of 
process, equipment and procedures. The FAT is conducted on a random sample from the 
first production lot. [Ref. 10] 

b. Live Fire Test and Evaluation (LFT&E) 

Live Fire Testing was mandated by Congress in November 1986. The law 
stipulates that major acquisition programs may not proceed beyond Low Rate Initial 
Production (LRIP) until realistic survivability (or lethality for some systems) testing has 
been completed. This testing requires the Services to Live Fire test their weapon systems 
as early as possible before Milestone Ш. [Кеғ. 11] LFT&E is its own type of testing and 
while it is not truly DT&E, the Congress recognized the importance of keeping the Live 
Fire Program coupled closely to the development process and affirmed this by requiring the 
Director, Live Fire Testing to report directly to the USDA. [Ref. 12] LFT&E is also 


sometimes associated with OT&E. However, there are significant differences between 
LFT&E and OT&E. OT&E is further described in the section "Operational Test and 
Evaluation” of this chapter. Figure 1] highlights the main differences between LFT&E and 


OT&E. 
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Testin 









FIGURE 1. Live Fire Testing Versus Operational Testing 


c. Production Acceptance Test and Evaluation (PAT&E) 

The Production Acceptance Test And Evaluation (PAT&E) is conducted on 
production items to demonstrate that those items meet the requirements and specifications 
of the procuring contracts so production may continue. PAT&E also ensures that 
production line systems demonstrate the same performance characteristics and capabilities 
of the preproduction models. Such testing is normally conducted by the Program Office 
quality assurance section, often at the contractor’s plant and may involve operational users. 
(Кет. 13] 

2. Operational Test and Evaluation. 
The purpose of Operational Test And Evaluation (OT&E) is to assess operational 
effectiveness and suitability at each stage in the acquisition process. Operational 
effectiveness is a measure of the contribution of the system to mission accomplishment 


under actual conditions of employment. Operational suitability is a measure of the 


maintainability and reliability of the system, the effort and level of training required to 
maintain, support, and operate it, and any unique logistic or training requirements of the 
system. OT&E may also provide information on tactics, doctrine, organization and 
personnel requirements and may be used to assist in the preparation of operating and 
maintenance instructions and other publications. OT&E’s most important aspect is that it 
provides an independent evaluation of the utility of the system and the feasibility of 
employing it. [Ref. 14] 

For major systems, OT&E is normally planned and conducted by a major OT&E 
field agency located within the DOD component. This Operational Test Agency (OTA) 
must be separate and independent from both the developing/procuring agency and the using 
agency. The OTA is responsible for managing operational testing, reporting test results 
and providing its independent evaluation of the system being tested directly to the Military 
Service Chief or Defense Agency Director. [Ref. 15] Like DT&E, OT&E also occurs 
throughout the acquisition process. OT&E’s role in this process is described in the 
“Testing and the Acquisition Process" section of this chapter. Figure 2 highlights the major 
differences between DT&E and OT&E. [Ref. 16] 

3. Multi-Service Test and Evaluation 

Multi-Service Test and Evaluation is T&E conducted on a system being acquired 
for more than one Service. All Services involved participate with one designated as the 
lead Service. At the conclusion of Multi-Service T&E, each participating OT&E agency 
submits an independent evaluation through its normal channels. The Lead Service then 
prepares a single report that reflects the system's operational effectiveness and suitability 
for each Service. This report goes forward to the Defense Acquisition Board (DAB) for 


review, recommendations and decision. [Ref. 17] 
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FIGURE 2. Differences Between DT&E and OT&E 


4. Joint Test and Evaluation 

Joint Service Test and Evaluation is a specific program activity sponsored by the 
Office of the Secretary of Defense (OSD). JT&E Programs are not primarily acquisition 
oriented. Rather, they are means of examining Joint Service tactics, doctrine and systems' 
interoperability. JT&E provides information on system requirements or improvements and 
for force structure planning. There are both Joint Development T&E and Joint Operational 
T&E. Joint Developmental T&Es focus on obtaining information on system requirements, 
performance, reliability and other technical aspects. Joint Operational T&Es are conducted 


to obtain data pertinent to operational doctrine, tactics and procedures. [Ref. 18] 


D. TESTING AND THE ACQUISITION PROCESS 
Testing is critical throughout the life cycle of any system. Both DT&E and OT&E 
events occur throughout the acquisition process which consists of the following phases: 
e Concept Exploration and Definition - Phase 0 
e Demonstration/Validation - Phase 1 
e Engineering and Manufacturing Development - Phase П 
e Production and Deployment - Phase IN 
“ Operations and Support Phase IV 
Figure 3 depicts these phases and illustrates that the phases are separated by key 
decision points or milestones. These decision points occur throughout the program life 
when a decision authonty reviews a program and authorizes it to advance to the next phase 
in the cycle. The following section describes the acquisition process and testing event: 
normally occurring during the respective phase. 
1. Concept Exploration and Definition Phase - PHASE 0 
The defense system acquisition process begins with the submission of a Missior 
Need Statement (MNS) with the Service's Program Objective Memorandum (POM). The 
MNS documents major mission deficiencies (or improvement opportunities) in a Service; 
ability to meet its mission requirements. The Concept Exploration and Definition Phase 
(C/E) follows the Milestone 0 approval for concept studies. During the (C/E) Phase 
alternative approaches for satisfying the requirement(s) are investigated. Concept! 
Exploration/Definition phase assists in Selecting preferred alternative system concepts. 
technologies, and designs. Documents for the Milestone I review are the MNS, the 
Acquisition Decision Memorandum (ADM) which provides the exit criteria, the Operational 
Requirements Document (ORD) which delineates the qualitative and quantitative system 
parameters and the Test and Evaluation Master Plan (TEMP) which identifies the 


objectives, responsibilities, resources and schedule for the T&E effort. 
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a. DTO 


Early in this phase a review of historical tests and existing systems is 


conducted. Preferred alternative system concepts are selected and a draft Test and | 


Evaluation Master Plan (TEMP) is started. The TEMP defines test phases, schedules and 


resource requirements. Prior to the Milestone I decision, laboratory testing, modeling and ; 


demonstrate and assess the capabilities of key subsystems and components. [Ref. 19] 


р. ОТ 0 


The operational test agency (OTA) estimates military utility and assesses ` 


operational impact of candidate technical approaches. The operational test agency also | 


monitors C/E Test and Evaluation for future test planning. OT&E conducted during this 
phase supports developing estimates of the need for the new system, a sound physical 
basis for the new system, the system’s affordability and the impact of the system on the 
force structure. [Ref. 20] 
2. Demonstration/Validation Phase - PHASE I 
After the Milestone I decision, the program enters the Concept 
Demonstration/V alidation (DEM/VAL) Phase during which selected concepts, typically 


brassboard or early prototype, are refined through study and analysis. [Ref. 21] This 


| 
i 
) 


| 
| 


simulations are conducted by the contractor and/or the Government development agency to ` 


phase ends with the Milestone II decision to either enter into Engineering and | 


Manufacturing Development (EMD), conduct more research and development and delay the 
decision or terminate the program. Documents of particular interest to the T&E manager at 
the time of the Milestone IT review include the ADM, an updated TEMP, the updated ORD, 
the Cost and Operational Effectiveness Analysis (COEA), which is a cost and operational 


analysis of the alternative systems, and the Development Baseline. 


а. ЮТ 1 
During this phase DT&E is accomplished to ensure that engineering is 

reasonably complete, demonstrate technical risk areas have been identified and can be 
reduced, identify the best or preferred technical approach and that the concept can meet 
operational requirements. DT l includes T&E of components, subsystems, and prototype 
development models. Testing also includes functional compatibility and interoperabilty 
with existing and planned equipment. DT conducted during this phase is most often 
conducted at the contractor's facility with Government oversight. [Ref. 22] 

b. OTI 

In OT I the OT&E agency prepares independent early operational assessments 
to identify the best design, indicate the risk level of performance for this phase of 
development, and estimate potential operational effectiveness and suitability. The 
operational aspects of the technical approaches is examined and information on tactics, 
doctrine, organization, personnel requirements and critical issues are identified. The OTA 
also identifies needed modifications or other issues that need to be resolved before the next 
phase is initiated. Typical operational and support personnel are used to obtain an estimate 
of the user's capability to operate the system. The OT&E assessments provide a record of 
testing, an audit trail, test data, recommendations and conclusions. Testing normally 
includes components, subsystems, brassboard configurations or advanced development 
prototypes. [Ref. 23] 
3. Engineering and Manufacturing Development Phase - PHASE II 

The objective of the Engineering and Manufacturing Development (EMD) Phase is 
to design, fabricate and test a preproduction system that closely approximates the final 
product. This phase may include Live Fire Testing if required. The information from the 
DT and OT along with other documents such as the Updated TEMP, the Beyond-Low Rate 
Initial Production Report (LRIP) and a Live Fire Test Report (The Beyond LRIP Report 


and Live Fire Test Report are required by law of the Director, Operational T&E) provide 

the information to the decision makers for determining whether to enter production or not | 

and what level of production. [Ref. 24] Data obtained during EMD test and evaluation are 

used to assist in evaluating the system's maintenance training requirements and in | 

evaluating the proposed training program. Test results generated during EMD also support | 

the user in refining and updating tactics. [Ref. 25] | 
a. DT Il 

DT IT must demonstrate that engineering is reasonably complete, that all 
significant design solutions to problems are in hand, and that the design meets its required 
specifications within the range of environmental limits designed for the operational 
employment of the system. DT I] also must verify "fixes" from DT I and assess the 
Survivability, vulnerability and logistic supportability of the system. Vulnerability (or 
lethality) may require live fire testing. 

The final phase of DT IJ is the TECHEVAL. The TECHEVAL is the formal 
demonstration that the design meets specifications and it provides the major source of data 
for certification of readiness for the OPEVAL. The TECHEVAL provides information | 
relative to the technical performance of the system. Itis the qualification of components 
and an assessment of compatibility, inter-operability, vulnerability, lethality, 
transportability, etc. The technical evaluation also determines performance limitations and | 
Safe operating parameters, insures the effectiveness of the manufacturing process and 
confirms readiness for operational testing. [Ref. 26] Typical test models for this phase 
include pre production prototypes or pilot production models. 

b. OTI 
OT II is conducted to demonstrate performance of the program objectives. It 


estimates operational effectiveness and suitability as well as identifies operational 
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deficiencies. OT II is used to determine adequacy of publications and support equipment 
and to provide information to refine operations and support (O&S) cost estimates. 

The final phase of OT I] is the OPEVAL. The Operational Evaluation 
(OPEVAL) occurs a minimum of 90 days after the TECHEVAL. It assists the developers 
by providing information relative to operational performance, doctrine, tactics, training and 
logistical issues. It assists the decision makers on the overall suitability of the system to be 
delivered as well as influences either a low rate initial production or a full-rate production. 
The OPEVAL also assesses the user's viewpoint on the system’s desirability. [Ref. 27] 
Typical test models for this phase include preproduction prototypes or pilot production 
models. 

4. Production & Deployment - PHASE II] 

The objective of this phase is to produce and field the system. Production 
Acceptance Test and Evaluation (PAT&E) is conducted on production items to ensure the 
effectiveness of the manufacturing process, equipment, and procedures. Follow-on 
Operational Testing (FOT&E) may be conducted to verify operational effectiveness and 
suitability. [Ref. 28] 

a. DT IH 

After the Milestone If] (Production and Deployment) decision, Developmental 
Testing remains an integral part of the development, validation, and introduction of system 
changes undertaken to improve the system or to reduce life cycle costs. [Ref. 29] DT II] 
verifies corrections of deficiencies in the TECHEVAL, verifies specification compliance 
and completes any testing not completed during EMD. DT II is also used to conduct the 
major elements of the PAT&E. The PAT&E ensures that production line systems 
demonstrate the same performance characteristics of the preproduction models. PAT&E 


will continue throughout the production life cycle of the system Testing conducted in this 
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phase is conducted under controlled conditions, provides quantitative and qualitative data 
and is normally monitored or conducted by a Government representative. [Ref. 30] | 

р. ОТШ 

OT III is used to verify correction of OPEVAL deficiencies and to evaluate 

performance not tested during earlier tests. OT III is conducted on the OPEVAL model. 
with fixes. OT III takes the form of Follow-on Test and Evaluation (FOT&E) and is. 
conducted with production articles in operational organizations. This testing verifies the 
production system, tests operational effectiveness and suitability under realistic operational | 
conditions and demonstrates reliability and maintainability improvements. [Ref. 31] 


5. Operations and Support - PHASE IV 


The function of this phase is to ensure that the fielded system continues to provide 






the capabilities required and to identify the actions and resources needed to maintain} 
operational readiness and support objectives. This phase ends with a Major Modification 
Approval to identify the actions and resources needed to achieve and maintain operational| 
readiness and support objectives. The Major Modification Approval encompasses a review 
of a system's operational effectiveness, suitability, and readiness to determine whether} 
major upgrades are necessary or deficiencies warrant consideration of replacement. In 
preparation for this milestone the TEMP, and product baseline are updated to describe 
program Status, changes and issues. [Ref. 32] 
a. DT 

Development testing during this phase ensures previous test deficiencies are 
corrected. DT evaluates proposed production improvements, Engineering Change 
Proposals (ECPs), upgrades and determines if the resources are available to maintain 
readiness and support objectives throughout the system's acquisition life cycle. As the 
system completes its useful life, DT is used in an engineering aspect to help modify the 


system for new threats or with new technology or to help in system disposal. 
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b. OT IV 
A major function of OT during this phase is to evaluate post production 
logistic readiness and support and to validate effectiveness and suitability of modified 
systems. Follow-on Test and Evaluation (FOT&E) are used to assess logistics readiness, 
sustainability, and the implementation of the Integrated Logistics Support Plan (ILSP). 
Finally as a system approaches the end of its usefulness, OT monitors the system's current 
State of operational effectiveness, suitability and readiness to determine whether major 


modifications are necessary or deficiencies warrant consideration of replacement. [Ref. 33] 


Е. DT&E IN THE UNITED STATES ARMY 

This section describes the testing structure starting from the Office of the Secretary of 
Defense (OSD) down to the key players involved in Army DT&E. 

1. DOD Test Structure 

The organizational structure of the DOD concerning acquisition and testing is 
depicted in Figure 4. T&E oversight is performed by two offices: the Director, Test and 
Evaluation (DTE) and the Director, Operational Test and Evaluation (DOT&E). The 
Defense Acquisition Executive (DAE), a position held by the Under Secretary of Defense 
for Acquisition and Technology (USD (A&T)), performs the management of acquisition 
for the DOD. 

The DTE is the principal staff assistant and advisor to the USD (A&T) for T&E 
matters. The DTE is responsible for all DT&E. The duties of the DTE include: review 
major acquisition program documentation for DT&E implications, provide management and 
oversight of the major ranges and test facilities, and develop and implement the Live Fire 
Test Program. 

The DOT&E reports directly to the Secretary of Defense (SECDEF) and has 


special reporting requirements to the Congress. The DOT&E's responsibility is to provide 
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unbiased insight into the operational effectiveness and suitability of new weapon systems. 
The duties of the DOT&E include approving test plans on major systems prior to OT&E, 
approval of OT&E funding for major systems and providing the SECDEF and the 
Congress with the Beyond LRIP report. [Ref. 34] 
2. Army T&E Structure 
The Army management structure for T&E is illustrated in Figure 5. The Under 
Secretary of the Army is the Army Acquisition Executive (AAE). The AAE is responsible 
for all acquisition T&E (operational and developmental tests) planning, programming, 
budgeting, developmental test policy and oversight. The AAE performs these duties with 
the assistance of the Assistant Secretary of the Army, Research, Development, and 
Acquisition (ASA/RDA). The ASA/RDA is organized to provide technical assessments and 
program evaluations. He resolves acquisition issues whenever possible and makes 
recommendations to the AAE on the acquisition of weapon systems. The Deputy Under 
Secretary of the Army for Operations Research (DUSA(OR)) 15 chartered to supervise all 
Army T&E policy and has oversight for all Army T&E. [Ref. 35] 
3. Army DT&E 
The U. S. Army Materiel Command (AMC) is responsible for the management of 
DT&E. Under AMC the Test and Evaluation Command (TECOM) has the primary 
responsibility for conducting developmental tests for the Army and the Army Materiel 
Systems Analysis Agency (AMSAA) conducts test analysis and evaluation on major 
systems. TECOM may be designated as the evaluator for non major systems. The 
Structure of AMC and TECOM is shown in Figure 6. 
TECOM is responsible for: 
e Planning, executing and reporting the results of technical tests. Technical tests include 
Development Tests, Technical Feasibility Tests, Production Qualification Tests, Joint 
Tests, and contractor/foreign tests. 


e Providing test facilities and technical expertise in support of the T&E life cycle. 
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STRICOM 


JEFFERSON 
PROVING 
GROUND 


(TECOM responsibilities continued) 
e Maintaining the Army's T&E data base. 


e Maintaining the Army’s Major Range and Test Facility Base. 





e Researching, developing, and acquiring instrumentation and developing new and | | 
improved test methodology. | 


e Providing safety confirmations. [Ref. 36] 


F. OTHER DT&E ORGANIZATIONS IN THE ARMY. 

The testing process extends throughout the acquisition cycle of any system. Many 
agencies play a significant role in the testing process. The Program Management Office 
(PMO), the Tester and his facilities, the builder or Contractor, the evaluator, the Army 
Materiel Systems Analysis Agency (AMSAA) in the Army's case and the user or Combat 
Developer all are a part of DI&E. They meet formally through the Test Integration 
Working Groups (TIWG) and informally based on other factors such as critical issues or 
Program Management style. 

1. Program Management Office (PMO) 

The Program Manager (PM) is ultimately responsible for all aspects of the system 
development, to include coordinating the total T&E program. The PM normally has a 
deputy or assistant whose functions include the overwatch of testing as well as writing or 
inputting into various test documents and reports. The PM and his office are responsible 
for writing numerous reports and plans such as the Test and Evaluation Master Plan 
(TEMP). The input to the TEMP is normally influenced by the Test Integration Working 
Group. 

a. Test Integration Working Group (TIWG) 

The PM charters and uses the Test Integration Working Group (TIWG) to 


help coordinate, plan, and discuss the testing and analysis effort. TIWG members include 
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representatives from the development agency, the user, both developmental and operational 
T&E agencies, logistics, analysis and training organizations. [Ref. 37] 
2. TECOM 
The U. S. Army Test and Evaluation Command (TECOM) is the Army's testing 
agency. TECOM as presented in Figure 6 has facilities throughout the U. S. as well as 
locations outside of the continental U.S. As sub components of TECOM, these facilities 
provide the people, equipment, and other resources to conduct various types of DT. 
TECOM through its various facilities is responsible for the planning, executing and 
Teporting the results of technical tests. Technical tests include Development Tests, 
Technical Feasibility Tests, Production Qualification Tests, Joint Tests, and 
contractor/foreign tests. TECOM is charged with maintaining the Army’s Major Range and 
Test Facility Base, maintaining the Army's T&E data base and researching, developing, 
and acquiring instrumentation and improved test methodology. 
3. Prime Contractor 
The Prime Contractor is the Company who is responsible to provide the 
Government with the needed product. The Prime Contractor plays a significant role in 
DT&E. The contractor conducts his own testing prior to Government tests and 
demonstrates that he is prepared to enter into Government conducted or at least 
Government observed testing. The contractor’s testing during the initial phases of the 
acquisition cycle is likely to impact testing conducted by the Government during the EMD 
Phase. The contractor may even conduct some DT, observed by the Government, within 
his facility. 
4. AMSAA 
The Army Materiel Systems Analysis Agency is AMC’s independent evaluator for 
the DT process often referred to as the "honest broker." AMSAA is responsible for the 


Independent Evaluation Plan (IEP) as well as advising the tester and the program office on 
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analytical issues, testing and test documentation. AMSAA's greatest influence is on the 
Statistical process controls of the tests such as sample size and confidence levels and test 
design. AMSAA conducts the analysis and/or evaluation of the testing and provides 
members to the TIWG. 


5. Combat Developer 


The Combat Developer is usually the “user” or the organization that represents the - 


user and identifies the need for the system being developed. In the Army this is usually a 
Training and Doctrine Command (TRADOC) basic Army branch such as the Field Artillery 
or the Armor School. These agencies develop the doctrine and training for their respective 
branches based on overall Army tactics, doctrine and guidance. They also help to 
determine the needs of that particular branch presently and into the future. These 
organizations provide the Mission Need Statements that may grow into development of a 
new system or modification of an old system. In some cases the user may be The 
Combined Arms Center (CAC) or The Combined Arms Service Support Center (CASSC). 
These agencies integrate the training, doctrine and needs of the various branches they 
represent and function as an overall user for a system needed by all the combined branches. 
The Combat Development agency is the organization where the acquisition process begins 


and where the final product arrives. 


G. POTENTIAL ISSUES AND PROBLEMS 

There are potentially many issues or problems that can affect Developmental Testing 
and many issues that are affected by DT&E. This chapter provided a basic history of 
testing in the Department of Defense, described testing within the acquisition process and 
the Developmental Testing function within the United States Army. The structure of the 
testing within the DOD and within the Army is effected by legislation, policy, both formal 


and informal guidance and a continuous reform effort. The management and conduct of 
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DT&E impacts upcoming milestone decisions and may hold the key to a program's success 
and continuation or its cancellation. The problems that occur in DT&E impact on cost, 
schedule and performance of the system and are very important to both the Government 
agencies and the contractors involved. The next chapter identifies those issues that are 


considered the most prevalent according to the major players in a number of programs. 
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ПІ. METHODOLOGY AND DATA SUMMARY 


A. INTRODUCTION 

This chapter presents the methodology used and data gathered to answer the primary 
and subsidiary thesis questions. An overview of the systems investigated is presented 
along with a summary of those data. Research investigation included a literature search of 
After Action Reports from Test and Evaluation Command (TECOM), lessons learned 
reports of major systems, General Accounting Office Reports, Congressional 
Subcommittee Reports, Developmental Test and Evaluation Reports and technical and 
professional journals and manuals. Interviews were conducted with program office 
personnel, program testers, analysis personnel, user representatives and contractors who 
have participated in the developmental testing of the major programs selected. Interviews 
were also conducted with personnel who had years of experience in the area of 


developmental testing. Interviews were conducted in person and over the phone. 


B. METHODOLOGY 

The focus of the literature search and the interviews was to address the primary thesis 
question. The primary thesis question was: 

What is the most significant problem in conducting developmental testing? 

1. Basic Interviews 

Interviews were the primary method of addressing the research question. A scope 

and limitation for the interviews were developed in order to organize the information. 
About a dozen systems were considered, this was reduced to seven. The main 
Government agencies dealing with DT&E were interviewed. These interviews included 


representatives from the Program Office, the tester, the analyst from AMSAA, a user 
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representative or the Combat Developer and the Prime Contractor. A basic set of questions 
was developed and reviewed by students and instructors with test and evaluation 
backgrounds. The initial format was general for all potential interviewees and looked at 
these areas: 

• What do you consider the primary problem or issue in Developmental Testing? 

e What can your agency or any of the others you work with do about the problem? To 
ascertain the "work with" relationship of the various agencies and offices, the 
questionnaire also focused on: 

• How do you (your office) interface with the other agencies? Is this sufficient? 

The questions further evolved into five separate formats, similar overall, but 
tailored to the particular agency being addressed. For example a PM office was asked how 
a particular problem affected schedule or cost. AMSAA would be asked instead how the 
problem affected their analysis or reporting. In most cases the interviewee was given a 
draft of the questionnaire or allowed to thoroughly answer all questions before any direct 
questioning. A copy of the questionnaire for the Program Management Office is provided 
in Appendix A. 

2. Special Interviews 
Other interviews were conducted to gain insight from people with background and 
experience in DT&E. These people were supervisors, branch and division chiefs at the test 
facilities, at AMSAA, at TECOM and within the TRADOC System Management (TSM) 
Offices. Using the same questionnaire format and focusing on the primary thesis question 
these individuals were also interviewed. Again, they normally had time to prescreen the 


questions or responded in writing when time was limited. 


C. SYSTEMS RESEARCHED 
The systems researched were Acquisition Category (ACAT) I and II systems tested at 


United States Army Test and Evaluation Command (TECOM) facilities within the United 
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States. Seven programs under test or tested in the past 10 years were researched and 
analyzed through interviews and reports. The systems include the Abrams Main Battle | 
Tank Block II upgrade (M1A2); Anti Armor Weapon System Medium (AAWS-M),"the 
Javelin; Enhanced Position Location Reporting System (EPLRS); the Avenger, a 
mounted Air Defense system, the Kiowa Warrior, an armed scout helicopter; the Maneuver | 
Control System (MCS), a command and control (C2) system; and the Family of Medium | 
Tactical Vehicles (FMTV). These programs represent different types of systems from | 
electronic/data communications and software to major weapon systems. They also 
represent different types of developments, from system upgrades and Non-Developmental 
Items (NDI) to full scale developments. 
1. Abrams Tank Block II Improvement (M1A2) 
The M1A2 is the MIA] (main battle tank) with improvements referred to as the 
Block II Improvement Program. The improvements to the main battle tank consist of an 
Improved Commander's Weapon Station (ICWS); Commander's Independent Thermal 
Viewer (CITV); Position Navigation System (POS/NAV); and the core tank. The core 
tank includes the turret, hull, fire control electronic units, a data bus to interconnect the new 
mission hardware; dual stabilization of the gunner's primary sight head mirror; the Single 
Channel Ground Airborne Radio System (SINCGARS); Inter Vehicular Information 
System (IVIS); and onboard built-in test equipment. The M1A2 is designed to provide | 
Armor and Mechanized units with improved mobility, protection and both internal and | 
external C3]. [Ref. 38] 
2. Kiowa Warrior (OH-58 D) 
The OH-58D is a modernization of the OH-58A airframe. The modernization | 
included a four blade main rotor system, an advanced cockpit display system and a mast 
mounted sight to provide day/night targeting capability. Armament was added to some of 


these helicopters for a special mission and eventually resulted in a modification program to 
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for all OH-58Ds. The Kiowa Warrior was designed to provide reconnaissance, security 
and target acquisition functions. It is used with Divisional aviation, in support of armor 
assets as well as against threat armor as part of a "Tank Killer Team” and employed by 
special operations units. [Ref. 39] 
3. Family of Medium Tactical Vehicles (FMT V) 
The new Family of Medium Tactical Vehicles is being designed to replace the 
Army's aging fleet of Two and a half (2.5) ton and Five (5) ton vehicles. These types of 
vehicles are used for tactical mobility, supply and support operations. The vehicles will 
include a 2.5 ton cargo model, a 5 ton cargo model and special purpose vehicles such as 
tankers, dump trucks and wreckers. Its expected improvement over the current fleet of 
vehicles includes greater horsepower and speed, increased towing capability, higher 
reliability and a high commonalty of parts among the various versions. [Ref. 40] 
4. Enhanced Position Location Reporting System (EPLRS) 
EPLRS provides the Army with data communications and ranging information. 
EPLRS reports position location and identification data on ground and airborne units to the 
radio station operator in near real time. Its performance advantages include rapid response 
times and effective data throughput, Communications Security (COMSEC), resistance to 
Electronic Countermeasures (ECM), and Electronic Support Measures (ESM), low levels 
of mutual interference, transmissions for ranging measurement and freedom from voice 
data contention. EPLRS will normally be deployed in the Division and Corps areas and 
operated by Army Signal Corps personnel. (Ref. 41] 
5. Anti Armor Weapon System -Medium (AAWS-M),"The Javelin" 
The Javelin is a man portable antitank weapon system. It consists of a round, a 
Command Launch Unit (CLU), training devices and test equipment. The Javelin has the 
capability to defeat the current and projected armor threat in all battlefield environments to 


include electronic and electro optical countermeasures and the electromagnetic 
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environments. The system provides the gunner with increased survivability by having a 
greater range, a reduced signature and increased lethality. [Ref. 42] 
6. Pedestal Mounted Stinger (PMS), "The Avenger" 
The Avenger is an Air Defense Weapon that has the requirements of protecting , 


friendly critical assets and inflicting maximum attrition on threat aircraft. It consists of a | 








fire contro] unit module that includes a turret with vehicle mounted launchers and a heavy ` 
machine gun. The system is mounted on a High Mobility Multipurpose Wheeled Vehicle 
(HMMWV) but can be operated remotely. The Avenger has a Forward Looking Infrared 


|| 
| 





(FLIR) sensor, a laser range finder and an onboard computer which provides the gunner < 
with displays to engage the target, monitor the system, and receive and display command 
and contro] C2] data. The Avenger is designed to protect the rear areas of the Corps, the 
Divisions and Regiments. [Ref. 43] 
7. Maneuver Control System (MCS) 
MCS is a combination of hardware and software intended to provide commanders 
of all maneuver elements (corps through battalion/squadron and selected companies) a 
single command and control (C2) system. It is one of five C2 systems that make up the 
Army Tactical Command and Control System (ATCCS). It includes a Lightweight 
Computer Unit (LCU), Large Scale Printer Plotter (LSPP), Large Screen Display, Tactical 
Scanner (TACSCAN) and software. MCS will enhance decision making and 
synchronization among maneuver elements and as a part of ATCCS help integrate and 
coordinate other battlefield functions such as Artillery, Air Defense and support functions. 
[Ref. 44] 
8. System Categories 
The systems listed above could be broken down by various criteria. To enhance 


analysis and address the thesis question they were broken down by type of system, namely | 
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ground vehicle, aircraft etc. The systems were also broken down by acquisition ог 
development strategy such as upgrade or NDI. 
a. Type of System 
The systems above include one rotary wing aircraft, the OH-58D; one man 
portable missile system, the Javelin; three ground vehicle systems including a tracked 
vehicle, the MJ A2; a small wheeled vehicle with a Air Defense missiles, the Avenger; and 
family of wheeled vehicles in different configurations, FMTV. The systems above also 
include two communication/data and information systems, MCS and EPLRS. 
b. Type of Development 
From a development standpoint the M1 A2 Abrams 1s an upgrade to the M1A1 
tank. The Kiowa Warrior 1s also an upgrade to the OH-58A scout helicopter. Both the 
Avenger and the FMTV are considered Non-Developmental but for slightly different 
reasons. The Avenger is using mostly developed technology designed for the military use 
while the Family of Medium Tactical Vehicles is pushing the use of commercial 
components and parts. The Javelin is full scale development weapon system as are the 


software and electronic intensive MCS and EPLRS. 


D. DATA SUMMARY 

The data summary includes information from both interviews and literature searches. 
The interview data will be broken down by program and further broken by agency or 
office. In order to obtain answers beyond the "party line" some interviews were conducted 
under the premise that the interviewee by name would not be associated with a particular 
program or agency. Therefore the data are presented by system but will not identify which 
system specifically. The person or office making the response about that system will only 
be identified as a representative of that agency or office who played a role in the system's 


DT&E. While few people actually requested anonymity, a single name or the name of the 
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system would easily divulge almost every person's identity to someone who is familiar 
with the agency or system described. Each of the following sections will summarize the 
respondents’ answers to the questionnaire, specifically: 

¢ What do you consider the primary problem or issue in Developmental Testing? 

e What can your agency or any of the others you work with do about the problem? 

e How do you (your office) interface (work with) with the other agencies? 

1. System t 
a. Program Management Office 

This program management representative stated that one of the major 
problems entering a Developmental Test was meeting the test start milestone with all 
compliant system hardware, software and support to conduct the testing. Delays in getting 
to the test start point (caused by various factors) cause a test delay and "all delays impact 
cost." 

The program manager representative determined that to deal with this 
problem, the PMO, particularly the PM, require intensive proactive management at all 
levels. Issues need to be identified before they happen and alternative plans developed. He 
indicated that the tester "...needs to make sure resource requirements are on hand, plus be 
in a positive position to adjust to changes." Finally he noted that all the entities involved in 
DT&E need to be more proactive and timely with their input into the test planning and 
development effort. 

The program manager representative described the interaction between his 
office and the other agencies as adequate and recognized each of the other agencies as 
having an important role to play in not only testing but the entire development process. 

b. Tester 

This tester considered the scheduling of DT&E the primary problem. He | 

credited PM over optimism and the budget/funding process as the causes. He stated that by 
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the time a system 15 supposed to enter DT&E, events have occurred that have caused the 
test window to be reduced and or delayed. The tester may often be left to prioritize the 
various tests, getting as many in as possible (but not all) before a report is required and 
decisions need to be made. 

This tester asserted that it was the tester’s job to deal with the situation as best 
they can. If anything could be done about this it would probably be the PM being a little 
more realistic about when DT&E will occur and how long it will take. The tester also 
Stated that despite this and some other problems " overall, the system works, especially for 
full scale development systems." [Ref. 45] 

The tester for "system t" believed there was a good relationship among the 
agencies and a "fantastic working relationship" between his office and that of the PM. He 
attributed the system's overall success to this interface. 

c. AMSAA 

The analyst saw scheduling as the primary problem with DT&E. Specifically 
he noted that on this program as well as others there was usually not enough time to test, 
collect data, compile data, analyze data, make preliminary reports and briefings and final 
reports and briefings. His analysis team often found itself working with incomplete data 
sets from which they were expected to make final reports required by the schedule. His 
primary concern was that key decisions were often made prior to completion of final 
reports. 

To help alleviate this problem he believed that AMSAA should be more 
realistic when signing up to a schedule and find ways of reducing their internal processes. 
He said that his agency was addressing the problem by working on methods to shorten 
their response time. He thought that the best way for the PMO to deal with this issue was 
to consider all that is involved in the evaluation portion of Test and Evaluation. He said the 


testers do the best they can given the environment in which they must test. 


ВЕ 


The lead analyst for “system t" considered the interface between his office, the 











tester and PMO as good. There were regular TIWG meetings, monthly reviews, ad hoc 
meetings and special working groups. He said there was little interface between his office 
and those of user and the contractor but did not see this as an issue. 


d. User Representative 


-smsa --.. 


The user representative for "system t" indicated that the most significant 
problem with DT&E was that the PM pushed the schedule rather than product readiness. 
He believed this problem was a result of the funding and budget cycles. The consequence 
of this schedule push was an early "bad name" for the system. He said that this system 
initially received a bad reputation because it went into a testing functionally unprepared for 
the test. 

The user representative thought that his office should have been more 
involved in the product design. He believed that the contractor and his office needed a 
closer and earlier interface. He indicated that the contractor's engineers still do not) 
understand the "real" operational environment. 

The user representative for this program interacts with the other agencies 
through action officers and through the PMO. The user representative believed that for this 
system the interface along with the TIWG is normally adequate but recommends more | 
frequent TIWGs and a more definitive TTWG schedule. 

e. Contractor 

The contractor believed obtaining the necessary resources to conduct a 
successful DT&E was the major problem in entering DT&E. Namely, needed activities 
were delayed, not accomplished or shortcuts taken. Also, resources such as funding were 
tight and schedules were compressed to the point that completing the test by a specific date 
became more important than conducting the test according to the test plan. Subsequently 


tests became more difficult to conduct and results harder to understand. The contractor 
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Stated these types of practices early in the program development tended to push problems 
into later phases where they were more expensive in terms of both cost and schedule. 

The contractor thought that all the agencies involved could help alleviate this 
problem. All agencies should provide better estimates of resources in terms of time and 
money. The PM should allow for contingencies and not assume perfect and complete 
Success at every step along the way. The contractor stated that long term stable funding 
would be the greatest help in overcoming schedule and resource problems, but that means 
legislative action. 

The prime contractor for “system t" was very positive about the 


contractor/Government PM relationship "for this program." He made the point that his 
experience with other programs between his company and the Army were not as 
cooperative. His interaction with the other agencies was mostly through the TIWG 
process. The contractor thought for this program the working relationships were good and 
provided an "easy flow" of information and good cooperation. 
2. System u 
a. Program Management Office 

The PMO representative said the most significant problem was that technical 
tests were conducted before the PMO and the contractor had sufficient time to do their own 
“checking out” of the system. This led to surprises during DT&E impacting test schedule 
and costs. The PM representative believed this happened because of inflexible budgets, 
changing Operational Requirements Documents (ORDs) and because the PM was locked 
into an inflexible success oriented schedule. 

He suggested that the PM, through management, should be able to work these 
issues. First, the PMO should get all the players involved early, including testers and 


evaluators, and concentrate on making realistic estimates. Then, the PM needs to ensure 
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that the ORD remains solid and realistic. He also said that PMs should view the tester as 
part of the team and not the "bad news messenger." 

The Program Management Office from "system u" was very pleased with the 
working relationships that were established within the program. Besides the formal TTWG | 
interface, the PMO had established other semi-formal groups to address numerous issues | 
including testing. The PM representative believed these working groups and their efforts 
enhanced the TIWG meetings as well as other aspects of the program. 

b. Tester 

This tester believed that the schedule was a major DT&E problem for this 
program. He said early involvement from the tester and the analysts is key to helping 
minimize this problem. The tester suggested that the PMO ensure testers and analysts are 
involved early on and that they actively scrutinize test and evaluation schedules before they 
are finalized. The tester for "system u" was satisfied with the interface and coordination 
that occurred for this program. 

c. AMSAA 

The AMSAA representative for "system u” identified the changing of software 
/nardware requirements as the biggest problem or issue in entering Developmental Test. 
Requirement changes often occurred after estimates were made and therefore affected the 
test plans and analysis. This changing test environment reduced confidence in the tests and 
impeded analysis. 

The AMSAA representative indicated that his agency can help with this 
problem by working with the PMO to help identify problems early in development. He 
also stated that AMSAA, the testers and the contractors must do a better job of controlling 
test costs. 

The analyst for "system u" said the interface between his office and that of the 


PM started as adversarial but improved over time. He believed he had a good working | 
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relationship with the tester and with the contractor. The analyst was not satisfied with the 
contact that he had with the user representative and believed that the analysis people should 
obtain more feedback from the troops. 

d. User Representative 

This user representative thought that the major problem in entering a DT&E 
was the flow of critical paper work. Namely, he pointed out those documents (TEMP, 
Detailed Test Plan) which are needed to make things happen. When these documents are 
late, it impacts and often delays the test schedule. The user representative believed for their 
part they should concentrate more on the war fighting capabilities and enhancements rather 
than technical issues. This may help reduce the paper delays. 

The user representative stated that his contact with the various agencies was 
frequent and provided for a good flow of information. The only interface that did not have 
regular communication was that with AMSAA. There was only minimal contact with 
AMSAA and it was usually formal in nature. He thought the TIWGs provided an adequate 
single forum to bring the key players together. 

e. Contractor 

The contractor cited unanticipated problems that delayed test completion 
within schedule and increased test costs as the major issues for DT&E. He ascertained that 
these occur because proposed estimates for cost and schedule usually assume no technical 
difficulties will be encountered. Subsequently overly optimistic estimates become the 
standard to meet. 

This contractor representative saw ways for his office, the PM and the Tester 
to address this issue. One way in which the contractor believed that his company could 
help deal with this problem was to use actual schedule information from historical records 
to create more accurate future test estimates. He said the tester also needs to track test 


program costs and schedule variances and document these historical data so that it can be 
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used when estimating future testing. He said the PM should identify potential problems to 
be encountered during testing and formulate contingency plans with cost and schedule 
impact acknowledged in the original estimates. | 

The prime contractor for "system u" noted that the contract office and the 
Government PMO had a very good working relationship. The interface with other agencies | 
like AMSAA and the test facility was more formal, less frequent, but sufficient. | 
Concerning DT&E, he does not recall involvement with the Combat Developer. 

3. System v 
a. Program Management Office 

The program representative for "system v," an NDI Program, identified the 
primary problem in conducting DT&E as the belief that the PM has plenty of money for 
test. He stated that testers and analysts tended to want to test extensively to reduce their 
risk and increase their confidence. However, the PM, like all PMs, had a limited budget 
and it was for more than testing alone. He attributed the problem to the acquisition process 
and the congressional funding system. 

Because the PM representative determined the root of the problem to be the 
acquisition process, he suggested that the problem was beyond the scope of the agencies 
and offices targeted for research. He did suggest that the PM bring in all the key players 
early, particularly AMSAA. 

The PM representative described his working relationship with the other 
agencies as regular and productive. Most of the interface is formal but gets more “нн 
and frequent as major tests or milestone reviews approach. 

b. Tester 

The tester indicated that the pnmary problem in conducting Developmental 

Testing was reactive involvement by the PMO instead of proactive involvement. The tester, 


believed that the PMO and the contractor often saw DT&E as an area to maybe save some 
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time or funds. The PM failed to put emphasis on DT&E until after something went wrong 
and both cost and schedule were negatively impacted. 

The tester believed that the test community as a whole should educate PMs to 
the various test capabilities available. Also, testers should demonstrate that the test facilities 
are more flexible than ever in packaging programs for the PM. He stated the main action a 
PM can do to avoid such a problem is to get the tester and the analysts involved early. The 
analysts for their part need to become flexible in packaging the analysis and evaluation. 
The analysts (AMSAA) must also realize that money no longer exists for huge sample sizes 
and that other methods are needed to analyze and design tests. 

The tester for "system v” believed his interface with the other agencies was 
adequate and was particularly good with AMSAA. Face to face coordination is easily 
achieved due to the close proximity to most of those agencies. He stated that good 
communication between the tester, the PM and AMSAA is important for successful DT&E. 

c. AMSAA 

The AMSAA representative for "system у" described the primary issue in 
DT&E as the Non-Developmental Item (NDI) status of the system. It was assumed for 
NDI programs, because they are "non developmental," that testing and analysis would be 
faster and easier. However, any type of a problem during DT&E or any other area in such 
a program can be costly. Problems in DT&E brought public scrutiny and possibly 
jeopardized the entire system development. This system, although NDI, still required 
extensive testing and data collection. The task of data reduction, analysis, and report 
preparation was reduced to a shorter time frame because of the NDI status. 

The analyst suggested some things that could be done by the various players 
to mitigate this problem. The AMSAA representative said that AMSAA is using different 
methods of analysis and evaluation to try to speed up the evaluation process. These 


methods include the physics of failure and the reliability growth model. The physics of 
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failure is the method of using more current electronic failure analysis instead of standards 
and specifications derived from early electronic hardware. The reliability growth model | 
tracks the increasing reliability of a system through its development and projects and plans | 
for levels of reliability at system maturity. The analyst stated that the PM needs to | 
recognize how non developmental a system truly is and develop realistic schedules. The | 
PM should also ensure that NDI is not automatically associated with easier testing and | 
evaluation. Finally, the analyst thought that the NDI test environment requires AMSAA, | 
the tester, the PM and the contractor to have a "team" approach to the test. 

The analyst from AMSAA for "system v" was very pleased with the working | 
relationships that had been established among the various agencies. The communication 
with both the test facility and the PMO was positive and frequent. There was little | 
interaction with the user representative and the contact with the contractor was limited to | 
formal forums such as scoring conferences or TIWGs. 

d. User Representative 

The Combat Developer addressed the changing of program timeline as the 
major problem on entering a Development Test. Specifically he referred to the compressing 
of the DT&E schedule and an unplanned DT&E and OT&E overlap in order to make up lost | 
schedule time. For example, the PM and the contractor prepared for OT I, conducted DT I 
during the preparation and tried to correct DT I deficiencies before the start of OT I. The | 
Combat Development Office believed that this problem "was driven by the desire to always 
present the program in a positive light (otherwise risk funding cuts)." [Ref. 46] 

The user representative office suggested the best way to deal with the issue | 
internally was to stake out a performance issue such as reliability and stick to it. This gives | 
the PM at least one solid perspective as to the system's readiness for test. The user | 
representative also said the PM should realistically assess performance and system | 


readiness based on the user's requirements and the system’s performance, not on "what it 





will take to get to the next hurdle." As for the other agencies and personnel involved, the 
user suggested that they need to be prepared to make the tough decisions, e.g. stop and fix 
the test process if needed. 

The user for "system v" said the interface with the other agencies is generally 
adequate but requires more intense coordination. He stated that the various conferences 
and working groups both formal and informal normally achieve their intent but often fail to 
resolve major conflicts. 

e. Contractor 

The contractor considered the biggest problem or issue in entering 
Development Testing the question of "how well a system made of many commercial parts 
will perform under rigorous military testing?" The problem stems from the Government's 
desire to have non-developmental systems yet maintain military specifications and 
Standards. In some cases the commercial parts cannot hold up to the stringent military tests 
and standards. 

The contractor suggested that everyone, especially the PM should be more 
aware of the complexities in buying commercial items for military use. Tradeoffs have to 
be made when buying under the commercial use concept (time versus Cost versus 
performance). He believed contractors should challenge the various military specifications 
and ascertain if a lesser performance level would be acceptable. The tester, who has 
knowledge and experience should be involved in contract specifications review for (NDI) 
contracts. Finally, the contractor recommended that the Combat Developer should help 
assess tradeoffs and delete unnecessary requirements. 

The Contractor for "system v" described different levels of interaction among 
the agencies. There was regular communication between the contractor's program office 
and the Government program office. There was also regular interface between the 


contractor and both AMSAA and the tester. The contractor had support personnel at two 
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test facilities attend meetings and facilitate good communications. The interface with the 
Combat Developer was less frequent but increased as the system prepared for another test. | 
4. System w 


a. Program Management Office 


жек РЦ 


The program representative stated that his major problem іп conducting ОТ&Е. 
was that you test regardless of how ready you are, "...it's a mark on the board you mie 
meet." The office also said there needs to be more control over the testing. Some tests are 
conducted to satisfy an evaluator's need but may add little to the overall analysis of the 
system. AMSAA particularly is not required to test prudently. All this non value added 
testing just makes completing DT&E within a very optimistic schedule that much harder. 

The Program representative suggested some actions for the various offices to 
deal with these problems. First, the PM should have everyone involved early -- about the 
time of the Statement of Work (SOW). The test and evaluation participants need to reduce | 
and justify tests, and budgets accordingly. He also stated that test facilities have recently 
become aware of this situation and have responded, but the evaluators were not as 
responsive. The PMO believed the Combat Developer should play a more definitive role 
in testing. A strong combat development office, that knows the system's background and 
the doctrine, could help decrease testing that adds no value to the system. | 

The program office responded that they had a good interface with the other 
agencies. They believed the formal interface of the TTIWG was good but attributed the 
positive working relationship between agencies to the fact that all the agencies were brought 
in early. 

b. Tester 

The tester for "system w" regarded the "lack of concrete requirements" as the 

most significant problem in conducting DT&E. This led to difficulties in test planning and 


resulted in schedule and cost overruns. 
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The tester admitted that his office could do better at coordinating the test effort 
with the PM, AMSAA, and the Combat Developer, but that the PM is the one who must 
bring these groups into harmony concerning the test requirements. As for the Combat 
Developer, the tester said he needed to define the requirements, learn and understand the 
System and appreciate the impact that changing requirements have on the test process. 
AMSAA should also try to better understand the system under test. 

The tester was pleased with the interaction and working relationships with the 
other agencies. The one exception he noted was that his interface with the Combat 
Developer was limited to the formal meetings such as TTWGs. He said that the working 
relationship was good but needed to be more frequent. 

c. AMSAA 

The analyst from AMSAA indicated that the biggest problem with DT&E was 
its uncertainty. That is, did the tester have adequate control to complete all testing on 
schedule and within budget? He further explained that some tests were not performed due 
to lack of time, funding or both. This creates "data voids" and makes the evaluation more 
difficult. 

The AMSAA representative believed that all the agencies can help at least 
mitigate the problem, but also asserted that it will take legislative action to get testing "event 
driven rather than schedule driven.” He stated for AMSAA's part, they should prionitize 
testing and work closely with both the PM and tester to design tests that fit within the given 
schedule and budget. The analyst also suggested that the PM fund the tester "as needed” 
rather than by fiscal year. The tester must learn more about the system under test and try to 
anticipate potential test control problems. The contractor should work closer with the tester 
to integrate the test item and the instrumentation. Finally, the analyst said that the Combat 


Developer should provide more explicit guidance on test set up and installation procedures. 


47 


The AMSAA representative concluded that the interface with the other 









agencies was normal and in most cases sufficient. The TTWG, the most common forum, 
was adequate for keeping the test community abreast of the status and latest developments 
in the program, but lacked in solving detailed, lower level issues. | 

4. Contractor | 

The contractor for "system w" said that the biggest problem in coni 
DT&E was the lack of adequate coordination between the tester and the c 
especially when integration checkout was needed between the test instrumentation and the 
-= system under test. This can severely impact both schedule and cost if restarts and retests 
are needed. | 

The contractor indicated that he is limited to bringing the issue to the attention 
of the PM and explaining its potential effect on the test and test data. He believed closer 
coordination with the tester and a better technical understanding by the tester would help 
alleviate the problem, but the PM needs to influence such a relationship. He also suggested 
that the Combat Developer establish realistic and unchanging requirements that will give the 
other agencies a foundation to develop tests and evaluation plans. 

The contractor described his interface with the other agencies as regular and 
sufficient. Contact with the PM was both formal and informal and contact with AMSAA 
and the tester usually in relation to reviews or technical documentation. He said there was 
very little contact with the Combat Developer. 

5. System x 
a. Program Management Office 

The Program Management Office stated that "The major problem entering this 

developmental test program was the compressed test schedule." The test schedule was laid 


out with no flexibility and testing continued throughout the program. When problems with 
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design or other areas were encountered, delays and slips occurred. The slips in turn 
delayed and or compressed DT&E as well as affected other areas such as OT&E. 

The Program Management Office had recommendations for the various 
agencies in addressing this problem. For the Program Management Office itself, the 
representative believed the PM must recognize the importance of the test schedule early and 
make a concerted effort to hold the line on design reviews, hardware delivenes and costs. 
The PM representative saw the tester as being left to complete testing within many 
constraints. He suggested that the tester try to be innovative and find ways to expedite 
testing. He believed the contractor could help by delivering hardware on time and 
committing sufficient resources to support the test. The PM representative suggested 
AMSAA should be open minded to problems, potential solutions and be able to make quick 
decisions. The Combat Developer also needs to respond to questions and issues in a timely 
manner. 

The PMO for "system x" characterized the working relationship with the other 
agencies as frequent, direct and both formal and informal. The PM representative 
considered this interface sufficient. 

c. AMSAA 

The analyst for “system x" indicated limited sample size as the major problem 
in developmental testing and evaluation. A limited sample size causes data analysis to be 
more difficult and increases risk. The analyst recognized that smaller sample sizes were 
becoming the norm as funding continues to decrease. 

He recommended that AMSAA rely more heavily on models and take part in 
building reliable models. AMSAA also should consider modeling and simulation in test 


design. 


49 


6. System y 
a. Program Management Office 

The Program Management Representative for "system y," an NDI program, 
stated that people have great expectations of NDI programs and so the testing schedule for 
such a program is very intensive. Trying to fit in all the testing that is required becomes 
difficult and is the primary DT&E issue for this type of program. | 

The PM representative for this system believed NDI type programs will | 
continue to be tested іп ап accelerated fashion and suggested ways that the PM and other | 
agencies could deal with this issue. He asserted that the PM should hold "conclusive" | 
TIWGs. That is, "Make the TIWG important and ensure that the other agencies send 
representatives that can make decisions and can speak for that office.” [Ref. 47] The PM 
also should have the tester and analyst on board early. He indicated that the tester should 
coordinate the test effort and start as early as possible. The analysts should accept some 
risk and the Combat Developer should develop a good set of requirements and then stick to 
them. Changing requirements severely impact the already intensive test schedule. The 
contractor should dedicate the nght people to the test and concentrate on putting them at the 
nght place during the key test events. 

This person indicated that the interface with the other agencies was good, but 
that the PM could improve the results through better management of the TIWG process as 
previously cited. 

b. AMSAA 

The analyst considered the use of contractor test plans and test data for 
evaluation and analysis as the major issue. Due to shrinking Government resources, the 
use of contractor data is becoming more common. 

To address this problem the analyst suggested that AMSAA should be more 


vigilant in reviewing test information provided by the contractor and the contractor should 
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be more receptive to some of the unique oversight required by the Government. The 
analyst also stated that the PMO should build safeguards into the TEMP to deal with using 
contractor testing and data. Finally the analyst concluded that even if the Government does 
not conduct the test in a certain case, the expertise of the tester will still be needed as will 
some of the Government test facilities. 

The analyst for this system believed that the working relationship between her 
office and that of the other agencies was sufficient. The contact between the various 
groups was conducted through both formal and informal means and conducted frequently. 
The analyst noted that the proximity of AMSAA to TECOM Headquarters was a positive 
contributor to the good exchange of information. 

c. User Representative 

The Combat Developer for this system considered the decreasing funds for 
RDT&E as the most significant problem in conducting DT&E as well as other types of 
testing. The decreased funding has caused compromised and reduced testing and increased 
risk. This reduced T&E and increased risk environment could be acceptable. However, 
typically when problems occur with a system, the program suddenly becomes a target for 
inquiry, funding reductions or even elimination because "...the Army failed to properly 
test." [Ref. 48] 

The Combat Developer conceded that decreasing funding 15 a reality and 
believed it will continue for the next few years. The best way that his office can deal with 
this problem is to actively participate in the TTWG process and insure the TEMP supports 
the ORD and the requirements in the ORD are valid and realistic. 

d. Contractor 
The contractor for "system y" stated that the major problem in conducting 


DT&E was the inconsistency between the ORD and the contract requirements. The 
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inconsistencies increased test cost and test time. Schedule was regained but at additional 


cost to both the Government and the contractor. 






The contractor suggested some actions to be taken if the inconsistency occurs | 
and also actions to prevent the problem in the first place. First, in order to resolve an | 
existing problem, the TTWG members did an extensive cross match of the contract, the | 
ORD and the TEMP. This created an overall, though not complete, consensus among the | 
TIWG members. Additionally, continuous tracking of the test issues by the PM helped | 
resolve the problems. To avoid variation between the ORD, the contract, and the TEMP in | 
the first place the contractor suggested that the PM and the Combat Developer thoroughly 
review the contract and verify that it corresponds to the ORD. 

The contractor described the relationships between his office and the various 
agencies as both formal and informal and as sufficient. He further stated that he did not 
believe the program testing would have gone as well if they had strictly relied on the formal 
interface of the TIWG alone. The informal working relationships were a key to the overall 
test success. 

7. System z 
a. Program Management Office 

The Program Management Office representative for this program cited the 
Government's changing requirements as the biggest problem in conducting a DT&E. 
Changing requirements are difficult for any program test, but with software intensive 
programs it's "really tough." [Ref. 49] When unplanned and unfounded requirements keep 
coming, none of the documentation is solidified. The detailed test plan, the software test 
plan, the Independent Evaluation Plan (IEP) and the TEMP are all affected by the changing 
requirements as is the test schedule. 

The PM representative saw some ways to address this problem. First, he said 


that the tester had to be involved early and be kept up to date on system changes that could 
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affect the test. Next, the tester should realize the complexities of software testing. Finally, 
the various agencies including the contractor need to make a team effort. He thought for 
"system z" that AMSAA took on an antagonistic role and that the contractor was not up 
front with bad news. A "team member spirit" might have helped avoided these problems. 
b. AMSAA 

The AMSAA spokesperson for this system believed that the major problem in 
conducting DT&E was that DT&E was a target for cutting costs. He suggested that this 
occurs because PMs are cost and schedule driven and that by the ttme DT&E rolls around 
many programs have cost and schedule overruns. PMs start looking for ways to save and 
they cut out some development type tests. Cutting tests impacts data, data analysis and 
creates more risk and development uncertainty. 

He recommended that the PM and the contractor should "realize the value of 
DT&E." They needed to "accept testing instead of seeing it as a burden." He also said that 
the contractor should be up front with potential problems, especially software problems. 

He described the interface between his office and the other agencies as 
changing. Originally it was formal, mostly TTWGs and teleconferencing, but this has 
improved by becoming more frequent and including other forums and less formal working 
relationships. 

c. User Representative 

The user representative for "system z“ regarded the lack of regulation and 
guidance for software testing to be the primary problem in conducting DT&E. This made 
the development and fielding of multiple versions of software extremely difficult. 

To resolve this problem the Combat Developer suggested that the PM office 
push for quicker fielding decisions, and the testers and analysts lobby for changes to the 


regulation and process for testing software. 
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The Combat Developer indicated that the interface between his office and the 
others was generally sufficient and that the TTWG was an adequate forum which served the 
purpose for which it was designed. He noted that the geographical location between the 
agencies; particularly between his office and the former contractor made that working 
relationship insufficient. 


8. Other Interviews 





A number of interviews were conducted with people who through their position 
and experience provided insight to the thesis questions. These people represented 
Supervisors, branch and division chiefs at a TECOM test facility, TECOM Headquarters 
itself and AMSAA. These interviews focused on the following: 

e What do you consider the primary problem or issue in Developmental Testing? 

e What can your agency or any of the others you work with do about the problem? 

a. TECOM I 
This project engineer said that schedule compression was the major problem 

in entering or conducting DT&E. He referred to the continuous pressure to conduct 
Developmental Testing in considerably less time than originally estimated. He believed that 
the funding and budget process drove this compressed schedule and caused major 
decisions to be made with only partial data. He also said that the budget process caused the 
PM to focus on budget and Initial Operational Capability (IOC), therefore risking system 
quality. He suggested the best way to deal with this issue is for the PM to understand early 
on what the capabilities are in the test and evaluation community and to work closely with 
the tester and analyst. 

b. TECOM II 

This representative from TECOM thought that "success oriented testing” was 
the major problem in entering or conducting DT&E. This unrealistic and optimistic attitude 


makes the tester a "bad guy or gal" when a test reveals a problem with the system. It also 
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fails to allow the contractor the time he should have to improve his design. Because 
problems are not planned for, it negatively impacts both cost and schedule when they do 
occur. PMs, contractors and everyone else should anticipate and identify potential 
problems. To alleviate this problem he thought that all the agencies should simply be more 
realistic about developing a test schedule. 
c. AMSAA I 
This senior analyst said the problem in entering developmental test was that in 
many cases we launch into testing with hardware or software that in reality is not ready for 
test. He believed that PMs do not always receive a realistic test status picture from their 
staffs. No one wants to deal with bad news, even potentially bad news. The analyst 
Suggested that PMs insure that their staff representatives for T&E have some test 
experience and coordinate closely with the testers and evaluators. 
d. AMSAA II 
Another senior analyst from AMSAA considered the schedule driven 
environment versus an event driven environment as the primary problem in conducting 
developmental testing. From the analysis point of view this causes problems with data 
collection, analysis and reporting. He feels this problem is not easily resolved by any 
agency or even all the agencies, because program funding is also schedule or calendar 
driven and all the participants know that. He suggested that early coordination, team work 
and realistic estimates by all the participants could minimize the effects of this problem. 
e. AMSAA IH 
The next analyst said the most significant problem was trying to obtain an 
adequate sample size, one large enough to analyze and yet not so large that it is cost 
prohibitive. He believed his agency must move to using more simulation and modeling and 


validate that information with a small number of actual tests. He also stated that more 
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positive incentives for contractors would be valuable. Rewarding the contractor early, for 
good designs that pass early basic testing would be an excellent investment. 
f. Tester I 


This tester stated the PM attitude about Developmental Testing was the most 





— 


significant problem in entering or conducting DT&E. He said that PMs are more worried | 


«= 


about the cost and schedule of testing than on using it as a tool. PMs often see DT&E as an 
area to try to make up cost or schedule overruns. This attitude is created by the process that | 
emphasizes getting everything right the first time. Testing often surfaces failures or | 
problems that the PM or contractor had not anticipated. Such failures can drastically impact 
a success oriented schedule. | 
The tester determined that to improve this situation testers should educate PMs 
as to their testing capabilities. Also, the decision makers need to be realistic in their 
expectations and let the "test, failure, fix, retest" model do its job. 
g. Tester ІІ 
Another senior tester saw the lack of early tester involvement as a primary 
problem in conducting DT&E. He believed even under an unrealistic schedule, the tester, © 
if involved early and kept informed could bring the test resources needed for the PM's 
requirements. Early participation by the tester and the analyst can help the tester customize 
the test program to satisfy the various data and analysis requirements as well as reduce 
costs. His bottom line was "bring the testers into the program early." 
h. Tester Il 
The next tester thought that the acquisition process itself was the major 
problem in conducting DT&E. The acquisition process causes PMs to be proponents of the 
system instead of proponents of the user. The emphasis 15 on program success instead of 


user factors. He believed it is the acquisition process that causes the unrealistic schedules, 


the adversarial relationships and systems that are fielded needing modes and retrofits almost 
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immediately. To fix this problem, the decision makers at very high levels (DA, DOD, the 
Congress for example) need to reward PMs that are critical and objective about their 
systems instead of fire them. (Ref. 50] 

9. Recent Studies and Initiatives 

Several recent studies have made recommendations that apply to the problems 
identified in this thesis. These recommendations include streamlining the T&E process; 
better risk management and increased acceptance of risk at all levels; testing smarter; 
acquisition reform; and early user involvement in the test process. 

Streamlining T&E or reducing the amount of actual testing and evaluation needed 
was a common theme throughout the studies. In the previous era (Cold War), test designs 
and test plans called for enough data generation to practically make evaluation a misnomer. 
Despite PM resistance to extensive testing, it usually still occurred. In the Post Cold War 
environment testers, evaluators and decision makers will no longer have the luxury of an 
unlimited amount of test data. [Ref. 51] The recommendations for reducing actual testing 
is to use modeling and simulation, integrate OT&E and DT&E where feasible and to 
increase decision risk analysis. The increase in risk apples to testers, evaluators as well as 
decision makers. Instead of a zero tolerance mode for development testing, a limited 
number of test criteria should be selected and an acceptance of test event risks outlined. 
[Ке!. 52] 

Throughout the studies and initiatives, the emphasis was on testing smarter and 
cheaper. The testing community in the Army; AMSAA, TECOM, OEC, and TEXCOM 
recognized the need for T&E efficiency and reduction and have started to formally meet, 
discuss and even implement some of the ideas. One example is the "improvement of 
requirements" determination. The emphasis is to develop realistic requirements that meet 
the user's needs and can be efficiently tested. The following example illustrates the concept 


of improvement of requirements. 
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A user had mandated that a system with a Mean Time Between Failure (MTBF) of 
80 hours increase to a MTBF of 150 hours at system maturity. After evaluating and 
scrutinizing the requirement it was determined that a MTBF of 150 hours made no 
significant impact on mission success but increased program risk and would take a lot 
of time to test. After a realistic analysis, the MTBF was increased to 114 hours at 
maturity. This took less test time, reduced program risk and improved the system 
within realistic terms. [Ref. 53] 
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An area mentioned by many respondents in the interviews was testing within the 
acquisition process itself. The respondents normally suggested ways to resolve problems | 


within the system believing the acquisition process too difficult to change. However, | 


A 


unlike other reform efforts in the recent past, there is anticipation that an opportunity truly” 
exists to improve the process. The Cold War is over and DOD resources are very limited 


and agencies like the GAO argue that this is an ideal time to change the system. 


Changes of the type needed will not come easily. They must be directed at the 
system of incentives that has become self-sustaining and very difficult to uproot. The 
incentives that motivate the participants must be realigned with better program 
outcomes. If we expect program sponsors to be forthright about program 
alternatives, costs and risks, such candor must be rewarded, and parochialism and 
undue optimism penalized. Ultimately, change will occur only through the collective 
action of the acquisition participants, particularly within the Department of Defense 
and the Congress, for it is their actions that dictate the incentives that drive the 
process. [Ref. 54] 


Early user involvement is another recommendation cited by the various studies. 

The studies state that the users can help the tester and evaluator focus in on those areas that 

are critical to mission success and system performance rather than just specifications. Early 

user involvement also should make users more familiar with the test environment so that 

they can help the tester and other decision makers determine the value and utility of future 

technologies. It should also give the user an appreciation for the test process and the 
impact of changing requirements. 

Today may in fact be the best opportunity the Government has ever had to” 

improve the overall acquisition process. Such an idea is politically popular and it appears 


more agencies than ever are looking into the "how" of changing the process. Any changes 
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to the acquisition process will likely affect testing and testing, in fact, continues to be one 


of the focus areas of the present reform effort. 
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ІҮ. RESULTS AND ANALYSIS 


A. INTRODUCTION 

This chapter discusses the results and analysis of the data presented in the previous | 
chapter. The focus for the analysis was on the primary thesis question: “What do you | 
consider the most significant problem in conducting Developmental Testing?" After | 
analyzing the agency responses it was determined that five problems areas were commonly 
noted across agencies. The responses were then categorized into one of the five common 
problem areas. The categories included: (1) Schedule problems, (2) Problems with the 
Acquisition Process, (3) Test Culture Problems, (4) Resources Management and (5) 
Changes in Requirements. The problems were analyzed by system, by agency as well as 
by the type of system and its development strategy. This analysis led to a unique finding in 
reference to a system's development strategy and discussed in the "Other Observations" 
section of this chapter.. The categories are explained below and Table I presents a 
simplified summary of the results. 

1. Schedule 

This category describes problems or issues related to test schedule and insufficient 

test time. The various respondents described these problems with "schedule crunch or 
squeeze," "lack of time for proper testing,” or "schedule push.” 

2. Acquisition Process 

The Acquisition Process category describes the responses that focused on testing 

problems that are a consequence of the overall acquisition process. These responses 
concentrated on the processes that create problems for DT&E as well as other areas. 


Common answers included: "the process creates unrealistic optimism and expectations," 
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“congressional funding does not allow for good long term planning,” or "the process 
overburdens both the Government and the contractor with bureaucracy." 
TABLE I 
INTERVIEW RESULTS 


een | чети Test Besmir] т їп 
АСЕМСУ Process Culture | Management| Requirements 
| K 


ХХХ 






Х - A single agency whose responses applied to that particular category. 
- Combat Developer 
** . Contractor 


3. Test Culture 

The next type of problem identified was described as test culture. This category 
consists of the responses which indicated that negative attitudes and stereotypes exist 
toward testing, testers and analysts. This "culture" is blamed for many of the problems 
including the adversarial relationships and inadequate communication and cooperation 
between the test community and other agencies. Responses included comments such as 
"late tester involvement," "lack of coordination due to adversarial role of the tester," 
“tester/analyst the bad news messenger,’ "DT - a place to make up lost schedule or dollars" 


or "DT as an inconvenience to the program ." 


6] 






4. Resources Management 
Resource management relates to problems noted by the respondents such as lack 
of funds, instrumentation, hardware or software. It also refers to the failure by one or 
more agencies to ensure that those same types of resources are at the right place, at the right 
time to conduct the proper testing. | 
5. Change In Requirements 
Change in Requirements is the final problem area in which responses well 
categorized. Problems of this nature occur when changes in the requirements in turn 
impact the TEMP, test conduct and or the evaluation. This problem was also mentioned by 


many of the respondents, but usually as an aside and not as the most significant problem. 


B. ANALYSIS 

This section summarizes the findings from the responses by category. The categories 
are further divided into findings across systems and findings across agencies. Each of 
these areas summarizes what the respondents determined as the cause(s) of the problem. 
Finally, for each category, recommendations are provided for problem minimization, 
avoidance or prevention. 

1. Schedule 

a. Findings Across Systems 
Schedule was the most frequent problem mentioned in the interviews. Five of 

the seven systems represented had at least one respondent describe schedule type problems. 
In addition, of those who stated that the acquisition process (next category) was the major 
problem, many pointed to the negative impact on schedule caused by the acquisition 
process. The process appeared to encourage unrealistic schedule estimates from all 
agencies. The respondents generally concluded that extremely optimistic management was 


the primary reason for schedule problems. Schedules were developed months, even years 
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in advance, always anticipating success along the way, but failing to take into account 
historical test information or previous test experience. If the test completion date could not 
be adjusted, test reduction and compression resulted in order to meet the schedule. 
р. Findings Across Agencies 
Across agencies, at least one representative from each agency described 
schedule as a significant problem. The testers and AMSAA believed that schedule 
problems were a product of unrealistic estimates. Testers and analysts were either not 
involved in early estimates or they signed up to an overly aggressive test schedule. The 
Combat Developers and PM offices cited over optimism and fear of funding cuts as the 
causes for schedule problems. The responding contractor in this category focused on 
estimates that assume no technical difficulties as cause of the problem. 
c. Recommendations 
The following is a summary of respondents’ recommendations to minimize or 
prevent schedule problems: 


e PMs should push for early involvement of all the participants including the testers and 
evaluators. 


• More realistic estimates should be made by all agencies involved with less optimism 
from the PM. 


• The PM should make the testers and evaluators part of the team and not the bad news 
messengers. 


e The Combat Developer should be actively involved in test planning from the 
beginning. 


e Historical information and data from previous tests should be used to better estimate 
future tests costs and schedule. 


Schedule was the most common problem mentioned in the interviews. The 
respondents generally cited PM optimism and unrealistic estimates as the cause of schedule 
problems. Overall early agency involvement and participation and realistic estimates were 


recognized as the best methods to prevent or minimize schedule problems. 
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2. Acquisition Process 
a. Findings Across Systems 

Across systems, six of the seven systems had at least one respondent describe 
the acquisition process as the most significant problem. The major causes for this problem | 
were the funding process and PM over-optimism. The annual control of funds forces PMs ° 
to be optimistic and show positive progress in cost, schedule and performance on a regular 
basis. If not, the PM faces possible funding cuts. 

PM optimism forces other agencies to plan and schedule based on the PM's 
extremely optimistic plans. Thus, testers and analysts sign up to try to meet aggressive 
schedules. The respondents concluded that these optimistic plans and schedules were 
unrealistic and based on meeting the schedule, not on historical test information or test 
experience. 

b. Findings Across Agencies 

Every agency identified the acquisition process as a problem. The PM 
representatives focused on the fear of losing funding and support as the cause of the 
problem. One tester indicated the acquisition process was the major problem. He believed 
the main cause was the current incentive system that rewarded PMs for being unrealistically 
optimistic. Two AMSAA representatives stated that the acquisition process was the major 
problem area and pointed to unrealistic early estimates as the cause. Two Combat 
Developers also determined the process was the major problem. One believed the cause 
was the layers of bureaucracy and paperwork. The other Combat Developer regarded 
inconsistencies within the process as the cause. The contractor also determined the 
inconsistencies as the cause of problems. For example: the military is told buy “off the 
shelf items,” but the items must still meet rigid requirements and standards that some 


"shelf™ items cannot possibly meet. 
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с. Recommendations 
The assumption for most of the respondents was that the acquisition process 
will not or cannot be changed or reformed enough to impact DT&E. The recommendations 
to improve acquisition process problems were based on that assumption and include: 


¢ PMs should push for early involvement of all participants including the testers and 
evaluators. 


¢ PMs should hold participative and conclusive TIWGs to address test plans and 
schedules. 


e The Combat Developer should be involved early and play a definitive role. 


• The analysts should utilize more efficient methods of evaluation, more modeling and 
simulation and accept more risk. 


e Senior decision makers should find a way to reward PMs who are critical and 
objective. 


The acquisition process was considered a major problem in conducting 
DT&E. The acquisition process was also cited as the cause of some of the other categories 
of problems identified in this thesis. Early and definitive involvement from the tester, the 
analyst and the Combat Developer were common recommendations for addressing this 
problem. 

3. Test Culture 
a. Findings Across Systems 

The Test Culture category had the third most responses overall. The 
representative causes noted for this problem included 1) the acquisition process itself, 2) 
lack of PMO understanding of test and analysis capabilities and constraints, and 3) the 
assumed reputation of testers and analysts as wanting to overtest. The respondents 
believed the acquisition process drove PMs to focus on cost and schedule and regard 
DT&E as an opportunity to make up time and money. Interviewees also indicated that PM 
Offices may not realize what the testers and evaluators can or cannot do within the 


constraints of the budget and the schedule. Therefore, unless the PMO involves the tester 
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and analyst early, PM offices could develop unrealistic test plans. Finally, some testers 
and analysts have earned poor reputations among Program Offices by conducting tests or 
pursuing additional data that a appeared to add no value to the process. This practice has 
caused increased costs and affected the credibility of testers and analysts. 
b. Findings Across Agencies 

The majority of the responses for this category were represented by test 
agencies. Four of the seven respondents in this category were testers, AMSAA and one 
contractor were also represented. The PM representatives and the Combat Developers as 
agencies did not respond in this category. The testers, three of whom were supervisors 


within TECOM, believed that test culture was root of many cost and schedule problems. 


They pointed to the acquisition process as the cause of this culture. The process | 


encourages the PM to move through testing quickly. If problems occur in testing, the 
testers and the analysts are usually the presenters of the bad news. Bad test news can mean 
rescheduling tests, may bring into question system need and validity from outsiders, and 
affect other cost and schedule issues for the PM. Two AMSAA representatives had 
responses that fit into this category. Both pointed to lack of funding and the funding cycle 
as the cause of the problem. The current funding system does not allow the PM efficient 
long term planning and in turn does not allow the testers and analysts to execute long term 
planning. One contractor representative believed that the acquisition process was the major 
cause of this negative approach to testing experienced in many PM Offices. He said that 
the process causes PMs to focus on cost and schedule and regard reducing DT&E reduction 
as an opportunity to make up for schedule and cost overruns. 
c. Recommendations 
The recommendations presented by the respondents for fixing, mitigating or 


preventing problems in Test Culture include: 
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e PMs should push for early involvement of all the participants including the testers and 
evaluators. 


e PMs should ensure that AMSAA, the tester and the contractor closely coordinate the 
test effort. 


e PMs and contractors should realize the value that DT&E provides to development. 


e Testers should educate PMs on their capabilities and demonstrate more flexibility in 
packaging test programs. 


е Testers should become more familiar with the systems under test. 
e Combat Developers should develop and stick to solid, realistic requirements. 


„+ The PM must make the testers and evaluators part of the team and not the bad news 
messengers. 


Test Culture problems were generally recognized by testers and analysts. The 
causes noted for this problem included the acquisition process itself, lack of PMO 
understanding of test and analysis capabilities and constraints, and the assumed reputation 
of testers and analysts as wanting to overtest. To prevent or minimize this problem most 
respondents determined that PMs should make the test community (testers, analysts) part of 
the team and the test community should better educate PMs, contractors and Combat 
Developers of their respective DT&E capabilities and limitations. 

4. Resources Management 
a. Findings Across Systems 

Resource management was mentioned as a problem by three of the seven 
systems. Respondents indicated that the causes of this problem included short term 
funding and limited resources (hardware and software) for DT&E. A system entering 
DT&E awaiting funding may not receive the resources in the lead time needed for proper 
test conduct. Lack of funding could delay test setup, delay instrumentation/equipment 
checks, cause inconclusive or even useless early test results and reduce needed test support 
personnel. Short term funding also causes PMs to desire and plan for perfect success in 


the test process. Anything other than perfect success could impact future funds. 
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Systems under development are often constrained by limited prototypes, test 
models, versions of software, and other components. Other required events of a system's 
development could cause these limited resources to be spread across the country and not at 
the test facility in time for proper test preparation. The lack of resources at the right place at 
the right time could severely affect test schedule, the test conduct or the evaluation and ' 
reporting process. 

b. Findings Across Agencies 

All agencies were represented in this category except the testers. The PM | 
representative indicated that resources management was the major problem and believed the € 
cause was the lack of aggressive management by the PMO. The two analysts from 
AMSAA believed that limited funding was the cause for this problem. The prime 
contractor pointed to reduced funding and compressed schedule as the cause of resource 
management problems. The Combat Developer believed the problem existed because of the 
increasing use of software in modern systems. Software testing, like the entire software 
management issue, lacks in information, experience and guidance. This is a new 
environment and creates many problems for testing as well as other functions. 

c. Recommendations 

The actions recommended by the respondents to resolve or at least minimize 

Resource Management problems include: 
e The PM should require intensive, proactive management at all levels. 
e PMs should plan for contingencies and not assume perfect success in the test process. 


e PMs fund testing to insure test resources are on hand when needed so that testers can 
be in a position to adjust to change. 


e All those involved in the DT&E process need to be timely with their input into the test 
plan. 


e Agencies should provide solid, realistic estimates of resources in terms of both time 
and dollars. 
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• Testers should become more familiar with the systems under test especially software 
intensive systems. 


Resources Management was another common problem area. The lack of 
proper resources at the right place, at the right time could severely affect the test and 
evaluation of a system. Short term funding and limited resources for DT &E were noted as 
the causes of this problem. The recommendations that addressed this problem included: 
fund testing to insure test resources are on hand when needed; tester familiarity with 
Systems under test, especially, software intensive systems; and incorporating more realistic 
estimates of test resources required by all agencies. 

5. Change In Requirements 
a. Findings Across Systems 

Change in (technical) Requirements is the final problem area in which 
responses were categorized. Four of those interviewed described this as the major problem 
Or issue in conducting DT&E. These four responses represented four different systems. 
They indicated that the causes of this problem were the lack of coordination and or 
communication between agencies and the lack of understanding of DT&E process among 
the Combat Developers. Lack of communication and coordination results in major 
documents such as the ORD, the TEMP, and the contract, not matching up with 
requirements. It causes difficulties in defining test requirements and makes test plans and 
conduct more difficult and expensive than onginally estimated. Combat Developers who 
may not be familiar with the test process may not realize the impact that a requirement 
change could have on the test and evaluation process. 

b. Findings Across Agencies 

Across agencies there was one response for each of the agencies except from 

the combat developers. None of the five Combat Developers or user representatives 


determined that change in requirements was the biggest problem in DT&E. This is notable 
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since these agencies are most likely to generate changes that impact the PM, the testers and 


the evaluators. The PM representative believed the lack of good communication and 





cooperation among agencies was the cause of the problem. The tester and the AMSAA © 
representative believed the lack of coordination and lack of understanding of the T&E ~ 
process on the part of the user representative was the cause. The contractor representative | 
pointed to both coordination and communication as the basis for this problem. 
c. Recommendations 

The recommendations for reducing requirement changes include: 
¢ The PM should insist that a solid, stable and realistic ORD be maintained. 

¢ PMs should hold participative and conclusive TIWGs. 


e The PM should establish a better working relationship among the agencies in defining 
test requirements. 


¢ The Combat Developer should appreciate the impact that changing requirements has on 
the system and the test process. 


e The PM should ensure that the major documents, to include the contract, are closely 
coordinated. 


Change in Requirements was the fifth category of significant problems. The 
respondents cited the lack of coordination and/or communication between agencies and the 
lack of understanding of DT&E process among the Combat Developers as the causes of 
this problem. Overall, the agencies believed that srong PM management of people, 
resources and critical documents was the best way to prevent problems associated with 


change in requirements. 


C. OTHER OBSERVATIONS 

Analysis was also conducted to determine if the type of system or its development 
strategy influenced the problems that occurred. This analysis used information gathered 
from the seven systems and did not include responses from the interviews conducted with 


supervisors at AMSAA and TECOM. Analysis by type of system (one aircraft, three 
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ground vehicles, one antitank weapon and two communication/data and information 
systems) revealed little relationship between the categories of problems and the type of 
system. This could imply that the problem areas identified in this thesis are generally 
applicable to Army systems regardless of type of end item. It also indicates that the 
recommendations that address these problems may be applicable to various Army systems. 
Responses were then analyzed by the development strategy of the systems against the 
categories of problems. The development strategies for the systems reviewed were: full 
development, major upgrade to an existing system and Non-Developmental ltem . The 


results are depicted in TABLE II. 


TABLE II 
DEVELOPMENT VERSUS PROBLEMS 


Е 
Development Process Culture | Management} Requirements 
a ep 
Development 







Upgrade 





X- A single agency whose responses applied to that particular category. 


TABLE II appears to indicate that overall, the type of development strategy may have 
minimal influence on the DT&E problems experienced by a system. However, there are 
two areas where a relationship may exist (shaded): 1) Schedule problems and upgrades in 


development and 2) the acquisition process and NDI developments. 
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1. Schedule and Upgrades 

Two systems, both representing upgrades in development, had at least three of the 
agencies for their respective systems indicated that schedule was the major problem in 
conducting DT&E. This indicated that a relationship may exist between schedule type 
problems and upgrade type developments. The cause for this relationship may be that 
“upgrades” are often seen as simply integrating new components and subsystems. DT&E | 
schedules are developed to focus on the upgrade. Upgrades however, may be extensive 
and incorporate the latest technology and the "simple integration” may prove more difficult | 
than anticipated. The early test schedule and planning for the upgrade probably does not 
anticipate the impact of new technologies and the significance of the integration on the old 
system. 

2. Acquisition Process and NDI 

Problems with the acquisition process and NDI developments also seem to be 
associated. There were two systems in the NDI spectrum of development. Most of the 
respondents, including both the PM representatives, for these systems indicated that the 
acquisition process was the major problem in conducting DT&E. The likely cause of this 
relationship is that NDI developments are often seen by senior leaders and the Congress as 
a kind of panacea acquisition model. NDI should be a more expedited process, to include 
DT&E. However, the acquisition process, as previously mentioned, already proliferates 
over optimism which is accentuated for NDI developments. This creates an environment of 
extremely high, unrealistic expectations for NDI developments. When these expectations 
are not realized within the original cost and schedule estimates, the system becomes the 


target of scrutiny and question. 
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3. Recommendations 
The recommendations that address these two relationships include: 


¢ PMs, as well as others, should avoid underestimating the DT&E process for ND] and 
system upgrades. 


e Historical test and analysis information and data from components and subsystems 
should be used to better estimate future tests costs and schedule. 


e PMs should hold participative and conclusive TIWGs to address test plans and 
schedules. 


e More realistic estimates should be made by all agencies involved with less optimism 
from the PM. 


e PMs should push for early involvement of all the participants including the testers and 
evaluators. 


Overall, the type of development strategy may have minimal influence on the 
DT&E problems experienced by a system. However, there are two areas where a 
relationship may exist: 1) Schedule problems and upgrades in development, and 2) the 
acquisition process and NDI developments. Both relationships may be the result of the 
high expectations of these types of development efforts. Many of the previous 
recommendations for schedule problems and acquisition problems hold true for these two 
relationships. In addition PMs, as well as others, should avoid underestimating the DT&E 


process for NDI and system upgrades simply because they are "supposed to be easier." 
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V. CONCLUSIONS AND RECOMMENDATIONS 


A. INTRODUCTION 

The primary purpose of this thesis was to explore and identify recurring problems in 
developmental testing in the United States Army, analyze the problems and make 
recommendations to prevent and or minimize these problems. As a result of this 
comparative analysis, it appears that Developmental Test and Evaluation is subject to many 
of the same problems that occur in the acquisition process. This chapter presents the 


conclusions and recommendations derived from the analysis of the previous chapter. 


B. GENERAL CONCLUSIONS 

This thesis concludes that five significant problem areas exist in conducting 
Developmental Test and Evaluation. In order of significance these problems are: 1) 
Schedule Problems, 2) Problems with the Acquisition Process, 3) Test Culture Problems, 
4) Resources Management Problems and 5) Problems with Changing Requirements. The 
thesis also concludes that the type of development strategy may influence which of these 
problems is most prevalent. Finally, the thesis concludes that the most recognized method 
to alleviate or prevent these problems is for the PM to involve the tester, the analyst and 


Combat Developer early in development. 


C. SPECIFIC CONCLUSIONS 
1. Schedule 
Schedule problems were the most common and the most significant problems in 
conducting DT&E. Schedule problems are caused by the acquisition process which 
encourages over optimism, unrealistic schedule estimates and emphasizes completing the 


test on schedule over conducting the test according to plan. The process may cause the PM 
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and his staff to develop early estimates without considering historical test information or the 
experience of the tester or analyst. 
2. Acquisition Process 

The acquisition process itself presents a significant problem to conducting DT&E 
as well as being a cause of other related problems. Nearly every agency addressed the 
acquisition process as a major problem. The causes for this problem included the funding 
process and PM over optimism. The funding process rewards PMs for being on schedule, 
under budget and meeting the criteria of the next milestone, but not for being critical and 
objective about their system and not for taking a user perspective. Over optimism by the 
PM in his planning and scheduling, forces other agencies in turn to Sign up to unrealistic 
plans that are based on meeting an aggressive schedule not based on the system's readiness 
for testing. 

3. Test Culture 

This thesis concluded that a negative test culture exists and this culture was the 
basis of many DT&E problems. PMs, their staffs, and sometimes contractors have a 
negative attitude toward testing, testers and analysts. The representative causes noted for 
this problem included: 1) the acquisition process itself, 2) lack of PMO understanding of 
test and analysis capabilities and constraints, and 3) the assumed reputation that testers and 
analysts require excessive testing. 

The acquisition process drives PMs to focus on cost and schedule and regard 
DT&E as an opportunity to make up time and money. PM Offices may not realize what the 
testers and evaluators can or cannot do for the PM unless the PMO involves the tester and 
the analyst early. Some testers and analysts have earned poor reputations among Program 


Offices by conducting tests that appeared to add no value to the process. 
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4. Resources Management 








Resources management of critical test assets was another major problem in | 
conducting DT&E. The causes of this problem included short term funding and limited 
resources (hardware and software). A system entering DT&E without adequate test 
funding may not receive the resources in the lead time needed for proper test conduct. Lack 
of funding could delay test setup, delay instrumentation/equipment checks, and reduci 
needed test support personnel. Short term funding also caused PMs to desire and plan for 
perfect success in the test process. Systems under development are often constrained by ` 
limited prototypes, test models, versions of software, and may be spread across the | 
country. The lack of resources can severely limit effective testing evaluation and reporting. 

5. Change in Requirements 

Changes in requirements were a major problem for DT&E. The causes of this 
problem were the lack of coordination and/or communication between agencies and the lack 
of understanding of DT&E process among the Combat Developers. Lack of 
communication and coordination resulted in documents such as the ORD, the TEMP, and 
the contract, not matching in terms of requirements. It caused difficulties in defining test 
requirements and made test plans and test conduct more difficult and expensive than 
originally estimated. Combat Developers, the agency where most changes come from, may 
not be familiar with the test process and may not realize the impact that a requirement 
change has on the test and evaluation process. 

6. Other Conclusions 
a. ‘Developmental Strategy 
This thesis also concluded that a system's development strategy may be 
related to the type of problems a system encounters. Two particular areas which reveal 
strong relationships were 1) Schedule problems and upgrades in development and 2) the 


acquisition process and NDI developments. The main cause for both these relationships 
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was that these types of developments tend to promote very high expectations among PMs, 
senior decision makers and other agencies. It has often been anticipated that there should 
be minimal problems in the DT&E of such developments (although the contrary is more 
likely). Therefore, when cost and schedule overruns occur prior or during DT&E, the 
senior decision makers, other agencies and even the Congress scrutinize and reassess the 
system. 
b. Early Involvement 

Early involvement of the tester, the analyst and the Combat Developer is 
critical to minimizing and or preventing DT&E problems. Having the PM bring these 
agencies in early to help estimate, plan, and coordinate the test effort was the most common 
recommendation made. This recommendation was observed across systems, agencies, and 


all categories of problems. 


D. RECOMMENDATIONS 

To improve Developmental Test and Evaluation, Program Managers, testers, analyst, 
Combat Developers and contractors should review and address the DT&E problems 
identified in this thesis. Specifically they should be prepared to address and account for 
problems involving: 1) Schedule, 2) the Acquisition Process, 3) Test Culture 4) Resources 
Management and 5) Changing Requirements, in that order. 

1. General Recommendations 

The PM should bring in all agencies for early planning, especially the tester, the 

analyst, and the Combat Developer. The PM’s DT&E effort should concentrate on realistic 
estimates of test cost and schedule, make the test community part of the team, aggressively 
manage test resources, and foster a working relationship between agencies that emphasizes 
cooperation and communication. The following is a list of general recommendations to 


prevent or minimize the problems identified in this thesis: 
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e PMs should push for early involvement of all the participants including the testers, 
analysts and Combat Developers. 


e The PM must make the testers and analysts part of the team and not the bad news 
messengers. 


e Agencies should provide solid, realistic estimates of resources in terms of both time ` 
and dollars. 


e Testers should educate PMs on their capabilities and demonstrate more flexibility in 
packaging test programs. 


e Combat Developers should develop and stick to solid, realistic requirements. 
2. Specific Recommendations 
The following recommendations are made to address each of the specific 
categories noted in the thesis. 
a. Schedule 


e Starting with the PM and his staff, more realistic schedule estimates should be made 
by all agencies involved 


e PMs should hold participative and conclusive TIWGs to address test plans and 
schedules. 


e Histoncal information and data from previous tests should be used to better estimate 
future test schedules. 


b. Acquisition Process 


e The analysts should not promote excessive testing and should integrate other and more 
efficient methods of evaluation including modeling and simulation. 


e Senior decision makers should reward PMs who are realistic and objective about the 
development of their system. 


c. Test Culture 


e PMs should ensure that AMSAA, the tester and the contractor closely coordinate the 
test effort. 


e PMs and contractors should realize the value that DT&E provides to their development 
effort. 





e Testers should educate PMs on their capabilities and demonstrate more flexibility in 
packaging test programs. 
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e The PM must make the testers and analysts part of the team and not the bad news 
messengers. 


e Testers and analysts should become more familiar with the systems under test to better 
understand, "What to test?” 


d. Resources Management 
е Тһе PM should require intensive, proactive management at all levels. 
e PMs should plan for contingencies and not assume perfect success in the test process. 


e PMs should fund testing to insure test resources are available when needed for proper 
test conduct. 


e Testers should become more familiar with the systems under test especially software 
intensive systems. 


e. Change in Requirements 
e The PM should insist that a solid, stable and realistic ORD be maintained. 


e The PM should establish a better working relationship among the agencies in defining 
test requirements. 


¢ The Combat Developer should appreciate the impact that changing requirements has on 
the system and the test process. 


• Тһе PM should ensure that the major documents, to include the contract, are closely 
coordinated. 


f. Other Recommendations 
The following recommendations focus on upgrade and NDI type 
developments. 


¢ PMs, as well as others, should avoid underestimating the DT&E process for NDI and 
system upgrades. 


e Historical test data and analysis information from components and subsystems should 
be used to better estimate future tests, costs, and schedule. 


¢ PMs should hold early TTWGs to address unique testing requirements, plans and 
schedules. 
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. AREAS FOR FURTHER RESEARCH 


The following areas should be investigated for potential benefit to the DOD: 


Developmental Strategy and Test Problems - One of the findings of this thesis 
was that the type of development strategy influenced the number of test problems a | 
system encountered. Further research on the effect of development strategy on test ` 


programs could provide insight for better tailoring of programs. 


Test Culture - Developmental Test and Evaluation could be significantly improved if 
the relationship between the test community (testers and analysts) and the PM Office 
were improved. Further research into the causes of this adversarial relationship with 
emphasis on preventing or minimizing its impact on testing could provide valuable 
information to the DOD. 


PM Incentives and the Acquisition Process - Researching the feasibility of 
incorporating an incentive system for PMs which encouraged them to be ‘event driven 
and user oriented" should be conducted. A meaningful and quantifiable rating and 
evaluation system for PMs that stressed good management techniques vice political 


maneuvering should be developed. 
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APPENDIX A (PROGRAM MANAGEMENT OFFICE QUESTIONS) 


Name: Location: 
Title or Job Date: 


Thank you for taking the time to meet/talk with me. I am conducting this research for my 
master’s thesis at the Naval Postgraduate School. The focus of my thesis is to identify 
problems in developmental testing from the perspective of the various agencies 
involved in DT&E. In my final product I hope to make some recommendations to correct 
the problems or at least minimize the impact of these problems for future programs. ] am 
examining a number of systems and interviewing the key players in the developmental test 
process of those systems as well as some others. My questions to you will primarily 
address your interface with these other agencies, what you perceive as the major problem in 
DT&E and how can you or the other agencies solve or alleviate the problem? I would also 
welcome any additional comments that you have about any aspect of the developmental test 
process. The information | collect will be confidential if you request. I will consolidate 
and summarize all interview data so that your name will not be identified in any way. 


PROGRAM MANAGERS 

1. How do you interface with these agencies: 
-Test facility 
-AMSAA 


-The contractor 


-The Combat Developer (user representative) 


2. Is this sufficient? 


5. What do you consider the most significant problem or issue in 


entering a Developmental Test? 


4. Does this problem effect the cost and or scheduling of the DT? 
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Э; 


What rules, regulations or policies exist that try to alleviate this problem? What 


rules, regulations or policies only feed the problem? 


Does the problem impact future testing - DT or OT? How? 


What could you the PM do to deal with this problem? 


What could be done by the Tester to deal with this problem? 


-Are they aware of the problem? 


What could be done by the Contractor to deal with this problem? 


-Are they aware of the problem? 


What could be done by the Combat Developer to deal with this problem? 


-Are they aware of the problem? 


What could be done by Agencies like AMSAA to deal with this problem? 


-Are they aware of the problem? 


What could be done by Legislative action to deal with this problem? 


What do you consider as other key problems or issues in Developmental Testing? 


What impact do these problems have on the cost and schedule of the test? 


What could be done by who and how to deal with these problems? 


82 





СОМЗЕС 


DA 

DAB 
DAE 
D&V 
DDT&E 
DEMVAL 
DOD 
DODD 
DODI 
DOT&E 
DT 

DTE 
DT&E 
DUSA (OR) 


ECP 
ECCM 
ECM 
EMD 
ESM 
EW 


FAT 
FLIR 
FMTV 
FOT&E 


GAO 


APPENDIX B 


(LIST OF ACRONYMS) 


Army Acquisition Executive 

Acquisition Category 

Acquisition Decision Memorandum 

Army Materiel Command 

Army Materiel Systems Analysis Activity 

Army Research Laboratory 

Assistant Secretary Of The Army (Research, Development & 
Acquisition) 

Assistant Secretary of Defense 

Army Tactical Command and Control] System 


Concept Exploration (Phase) 

Command Launch Unit 

Commander's Independent Thermal Viewer 

Cost and Operational Effectiveness Analysis (Report) 
Communications Security 


Department of the Army 

Defense Acquisition Board 

Defense Acquisition Executive 
Demonstration and Validation 

Director, Defense Test and Evaluation 
Demonstration and Validation Phase 
Department of Defense 

Department of Defense Directive 
Department of Defense Instruction 
Director Operational] Test and Evaluation 
Development Test 

Director, Test and Evaluation 
Development Test and Evaluation 
Deputy Under Secretary Army (Operations Research) 


Engineering Change Proposal 

Electronic Counter-Countermeasures 
Electronic Countermeasures 

Engineering and Manufacturing Development 
Electronic Support Measure 

Electronic Warfare 


First Article Test 

Forward-Looking Infrared 

Family of Medium Tactical Vehicles 
Follow-on Operational Test and Evaluation 


General Accounting Office 
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ICWS 
IVIS 


JT&E 


LCU 
КЕТ&ДЕ 
LRIP 
LSPP 


MCS 
MST&E 


‚ МО] 


OPEVAL 
OPES 
ORD 
OSD 
OTA 


PAT&E 
PM 
PMO 
РОТ 


R&D 
RDT&E 


SAE 
SINCGARS 
SOW 
STRICOM 


TACSCAN 
T&E 
TACOM 
TECHEVAL 
TECOM 


USD (A&T) 
WBS 


High Mobility Multi-purpose Wheeled Vehicle 


Improved Commander's Weapon Station 
Inter Vehicular Information System 


Joint Service Test and Evaluation 


Lightweight Computer Unit 
Live Fire Test and Evaluation 
Low Rate Initial Production 
Large Scale Printer Plotter 


Maneuver Control System 
Multi-Service Test and Evaluation 


Non-Developmental Item 


Operational Evaluation 

Army Operational Test and Evaluation Command 
Operational Requirements Document 

Office of the Secretary of Defense 

Operational Test Agencies 


Production Acceptance Test and Evaluation 

Program Manager/Project Manager/Product Manager 
Program Management Office 

Production Qualification Test 


Research and Development 
Research, Development, Test and Evaluation 


Service Acquisition Executive 

Single Channel Ground and Air Radio System 
Statement of Work 

Simulation, Training and Instrumentation Command 


Tactical Scanner 

Test and Evaluation 

Army Tank-Automotive Command 
Technical Evaluation 

Army Test and Evaluation Command 
Test and Evaluation Master Plan 

Test and Experimentation Command 
Test Integration Working Group 

Army Training and Doctrine Command 
TRADOC System Manager 


Under Secretary of Defense (Acquisition & Technology) 


Work Breakdown Structure 
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